What Is Statistical Learning?

Every supervised model on this site - linear regression, trees, neural networks - is an answer to the same question, posed in the same notation. Before comparing methods it is worth stating that question exactly, because it also tells you which part of your error you can hope to remove and which part you are stuck with no matter how good your method is.

A. The setup

We observe a response $Y$ and $p$ different predictors $X = (X_1, X_2, \dots, X_p)$. We assume there is some relationship between them, written in the general form

$$Y = f(X) + \varepsilon .$$

Here $f$ is a fixed but unknown function representing the systematic information that $X$ provides about $Y$, and $\varepsilon$ is a random error term, independent of $X$, with mean zero.

Statistical learning is the set of approaches for estimating $f$.

The estimate is written $\hat f$, and the prediction it produces is $\hat Y = \hat f(X)$.

B. Reducible and irreducible error

Suppose for a moment that $\hat f$ and $X$ are fixed. How wrong is $\hat Y$ as a prediction of $Y$? Following James et al., the expected squared error decomposes into two conceptually distinct pieces:

$$\mathbb{E}\big[(Y - \hat Y)^2\big] = \underbrace{\big[f(X) - \hat f(X)\big]^2}_{\text{reducible}} + \underbrace{\operatorname{Var}(\varepsilon)}_{\text{irreducible}} .$$

The first term is reducible: $\hat f$ is not a perfect estimate of $f$, and we can shrink that gap by choosing a better method or gathering more data.

The second term is irreducible, and this is the important one. Even with a perfect estimate - even if $\hat f = f$ exactly - the prediction would still be wrong by $\varepsilon$, because $Y$ depends on $\varepsilon$ and $\varepsilon$ is by definition not predictable from $X$.

Why is the irreducible error larger than zero? Two reasons, both worth internalising. First, $\varepsilon$ may contain unmeasured variables that would help predict $Y$ - since we did not measure them, no $f$ built on $X$ can use them. Second, it may contain genuinely unmeasurable variation: the same inputs on two different days simply do not produce identical outputs.

The practical consequence is a hard ceiling. $\operatorname{Var}(\varepsilon)$ is an upper bound on how good any model can get, and it is almost always unknown in practice. A model that appears to beat it is not a triumph; it is a sign you are measuring test error on data the model has already seen.

C. Prediction versus inference

There are two reasons to estimate $f$, and they pull in opposite directions.

For prediction, you only care that $\hat Y$ is close to $Y$. The estimate $\hat f$ can be a total black box - you will never inspect it - provided it is accurate.

For inference, you want to understand the relationship: which predictors actually matter, whether each one raises or lowers the response, whether a simple linear summary is adequate. Now $\hat f$ cannot be a black box, because the shape of the model is the answer.

This tension recurs constantly:

A model chosen purely for accuracy will often be one you cannot explain, and a model chosen for explanation will often leave accuracy on the table. Knowing which of the two you are doing is a prerequisite to choosing sensibly.

D. Parametric and non-parametric estimation

Broadly there are two strategies for producing $\hat f$.

Parametric methods reduce the problem to estimating a fixed set of numbers. You first assume a functional form - most simply, that $f$ is linear:

$$f(X) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p ,$$

and then only need to estimate the $p+1$ coefficients rather than an arbitrary $p$-dimensional function. That is an enormous simplification. The risk is equally clear: if the true $f$ is far from linear, no choice of coefficients will fit it, and the model is wrong in a way more data cannot fix.

Non-parametric methods make no such assumption and let the data determine the shape. They can fit a much wider range of true functions, but because they do not reduce the problem to a few parameters, they need substantially more observations to pin the shape down reliably.

E. Why flexibility is not free

It is tempting to conclude that flexible non-parametric methods are simply better. They are not, for two reasons.

First, as noted, they are hungrier for data. Second - and less obviously - a very flexible method can follow the noise $\varepsilon$ in the training sample as if it were signal. The fit to the data you have looks superb while the performance on data you have not seen degrades.

The figure draws schematic curves, not the polynomial fits below: as you raise Model complexity, training error falls steadily while test error turns back up.

The following demonstrates the point on data where the truth is known, so we can compare the fit against $f$ itself rather than against noisy observations:

Running it prints:

degree  1:  training MSE 0.1985   error vs true f 0.2465
degree  3:  training MSE 0.0542   error vs true f 0.0235
degree 15:  training MSE 0.0258   error vs true f 730.5159

Read the two columns against each other. Training MSE falls monotonically with flexibility - degree 15 fits the observed points best of all three, at $0.0258$. Yet its error against the true $f$ is $730.5$: not marginally worse than degree 3's $0.0235$, but roughly thirty thousand times worse.

The blow-up is worth understanding rather than just noting. A degree-15 polynomial fitted to 25 points has enough freedom to weave through the noise, and the wiggles it needs in order to do so become violent near the edges of the interval, where there are fewest points to constrain it. Evaluated on the dense grid, those boundary excursions dominate the average. This is a real and well-known failure mode of high-degree polynomial fits, not an artefact of the seed.

Meanwhile degree 1 shows the opposite failure: its training error is much worse than degree 15's, but it is honest about it - the error against the truth, $0.2465$, is about the same as its training error, because a straight line is too rigid to chase noise. It is simply the wrong shape for a sine wave.

Degree 3 wins on the only column that matters. This gap between "fits the sample" and "captures the truth" is the central phenomenon of the subject, and it has a precise decomposition.

Key takeaways

• Statistical learning estimates an unknown $f$ in $Y = f(X) + \varepsilon$.
• Prediction error splits into a reducible part, which better methods can shrink, and an irreducible part $\operatorname{Var}(\varepsilon)$, which nothing can.
• Irreducible error exists because of unmeasured and unmeasurable variation; it is a hard ceiling on any model's accuracy.
• Prediction favours flexible, opaque models; inference favours restrictive, interpretable ones. Decide which you are doing first.
• Parametric methods assume a form and estimate few parameters; non-parametric methods assume less but need far more data.
• More flexibility always improves training error, and beyond a point worsens accuracy on new data.

What's next

That last observation deserves an exact account rather than an anecdote. The reducible error itself splits into two competing pieces - one that shrinks as models get more flexible and one that grows - and their sum is what you actually pay. That is The Bias–Variance Tradeoff.