The Bias-Variance Tradeoff

What Is Statistical Learning? showed a degree-15 polynomial fitting its training sample better than any competitor while being catastrophically wrong about the underlying function. That was a demonstration. This article gives the account: the expected test error splits into exactly three pieces, two of which we control and one of which we do not, and the two we control move in opposite directions.

A. The decomposition

Take a test point $x_0$, and let $\hat f$ be a model fitted to a random training sample. The expected squared error at $x_0$, averaged over all the training samples we might have drawn, decomposes as

$$\mathbb{E}\big[(y_0 - \hat f(x_0))^2\big] = \underbrace{\big[\operatorname{Bias}(\hat f(x_0))\big]^2}_{\text{wrong on average}} + \underbrace{\operatorname{Var}(\hat f(x_0))}_{\text{unstable}} + \underbrace{\operatorname{Var}(\varepsilon)}_{\text{irreducible}} ,$$

where the bias is $\operatorname{Bias}(\hat f(x_0)) = \mathbb{E}[\hat f(x_0)] - f(x_0)$.

This is an equality, not an approximation or a heuristic. Each term means something concrete:

• Bias is the error from approximating a complicated real problem with a simpler model. A straight line fitted to a curve is biased everywhere, and no amount of extra data removes that.
• Variance is how much $\hat f$ would change if you refitted it on a different training sample of the same size. A method that swings wildly from sample to sample has high variance, and any single fit is unreliable.
• Irreducible error is $\operatorname{Var}(\varepsilon)$, the floor from the previous article.

Since squared bias and variance are both non-negative, their sum is non-negative, so expected test error can never fall below $\operatorname{Var}(\varepsilon)$. The floor is real.

B. Why the two terms fight

The general pattern: as a method becomes more flexible, bias falls and variance rises.

More flexibility means the model can bend towards the true $f$, so it is less systematically wrong - bias down. But it also means the model can bend towards the particular noise in this sample, so a different sample would produce a noticeably different fit - variance up.

Test error is the sum. Early on, flexibility buys a large bias reduction for a small variance increase and total error falls. Past some point the trade reverses and total error climbs. The best model sits where the two rates of change cancel.

The figure draws schematic curves in arbitrary units, not the $\sin(2\pi x)$ simulation below; its noise floor sits at 0.16, not the 0.09 used there.

C. Measuring all three terms

The decomposition is usually presented as theory because in practice you cannot compute it - you do not know $f$ and you have only one training sample. But in a simulation we control both, so we can measure each term separately and check they really do add up.

The setup: $f(x) = \sin(2\pi x)$, noise $\varepsilon \sim N(0, 0.3^2)$ so $\operatorname{Var}(\varepsilon) = 0.09$, training samples of 30 points, and a single test point $x_0 = 0.35$. We refit 2,000 times on fresh samples and watch what the predictions at $x_0$ do.

Running it prints:

true f(x0) = 0.8090,  Var(eps) = 0.0900

degree 1:  bias^2 0.2684   variance 0.0130   + noise 0.0900   = 0.3713
degree 3:  bias^2 0.0059   variance 0.0105   + noise 0.0900   = 0.1064
degree 9:  bias^2 0.0000   variance 0.0295   + noise 0.0900   = 0.1195

D. Reading the table

Every claim in section B is visible here as a number.

Bias collapses as flexibility grows. Degree 1 has squared bias $0.2684$ - a straight line simply cannot pass near $\sin(2\pi x)$ at $x_0 = 0.35$, and its average prediction over 2,000 fits was $0.291$ against a true value of $0.809$. By degree 3 the squared bias is $0.0059$, and by degree 9 it rounds to zero: on average, the flexible model is exactly right.

Variance grows as flexibility grows. It more than doubles from $0.0130$ at degree 1 to $0.0295$ at degree 9. The degree-9 fit is right on average but any individual fit is noticeably off, in a direction that depends on which 30 points it happened to see.

The sum is minimised in the middle. Total error runs $0.3713 \to 0.1064 \to 0.1195$. Degree 3 wins - not because it is best on either term individually (degree 9 has lower bias) but because it is the best compromise.

The floor holds. No configuration gets below $0.09$. The best total, $0.1064$, is only $0.0164$ above the irreducible noise, and that remaining gap is the reducible error still on the table.

A caution about the degree-9 row. Squared bias printing as $0.0000$ does not mean the model is unbiased everywhere - only at this particular $x_0$, and only to four decimal places after averaging 2,000 fits. Bias is a function of $x$; at a point near the boundary of the interval the same model would show substantial bias, for exactly the reason the degree-15 polynomial exploded in the previous article.

E. Does the decomposition actually hold?

The three terms should sum to the expected test error, which we can measure directly by generating fresh noisy observations at $x_0$ and averaging the squared prediction errors. Doing that alongside the decomposition gives:

The two columns agree to within a few thousandths - the residual difference is Monte Carlo error from using 2,000 simulations rather than infinitely many. The identity is exact; our estimate of it is merely very good.

F. What this means in practice

You cannot compute bias and variance on real data, because you have one sample and no access to $f$. What you can do is recognise their signatures:

Note the asymmetry in the fixes: more data reduces variance but does nothing for bias. Doubling the sample will not help a straight line fit a sine wave. If your model is biased, you need a different model, not a bigger dataset.

Key takeaways

• Expected test error decomposes exactly into squared bias, variance, and irreducible noise.
• Bias falls and variance rises with flexibility; the sum is U-shaped and the best model is the compromise, not the extreme.
• In simulation the terms are measurable: degree 3 won with total $0.1064$ against $0.3713$ (biased) and $0.1195$ (high-variance).
• Test error can never go below $\operatorname{Var}(\varepsilon)$.
• More data cures variance, not bias. Diagnose which you have before choosing a remedy.

What's next

The diagnostic table above needs an honest estimate of test error, and the training error emphatically is not one. Getting that estimate from the data you already have - without a separate test set you may not be able to afford - is the job of Cross-Validation and Resampling.