The Bias–Variance Bargain
Every model trades two kinds of error. The trick is knowing which one you're paying for.
Every prediction error breaks into two parts: the part that comes from the model being too simple to fit the truth (bias), and the part that comes from the model being too sensitive to the particular data it was shown (variance). You cannot, in general, drive both to zero at once. That is the bargain.
The cartoon
- A line through a curved cloud of points has high bias, low variance. Show it new data; it predicts about the same wrong thing.
- A degree-30 polynomial through ten points has low bias, high variance. Move one point and the curve writhes.
The sweet spot is a model just flexible enough to capture the signal and stiff enough to ignore the noise.
Why it matters
The bargain is the reason regularization works, why bigger datasets let you use bigger models, and why a held-out test set is non-negotiable. It is also why, when a model performs badly, your first question should be: am I underfitting or overfitting? The fixes are opposites.
Train error low, test error high → variance. Train error high, test error high → bias. Train error high, test error low → check your code.