Basis Functions and Piecewise Polynomials

Linear regression assumes the effect of a predictor is a straight line. When it is not, the usual instinct is to reach for a different algorithm. There is a cheaper move: keep least squares and change what you regress on.

The basis function idea

Choose a family of transformations of $X$, fixed and known in advance:

$$b_1(X),\ b_2(X),\ \dots,\ b_K(X),$$

and fit

$$y_i = \beta_0 + \beta_1 b_1(x_i) + \beta_2 b_2(x_i) + \cdots + \beta_K b_K(x_i) + \varepsilon_i .$$

Two familiar methods are instances of this.

• Polynomial regression takes $b_j(x) = x^j$.
• Step functions cut the range at $c_1, \dots, c_K$ and take $b_j(x) = I(c_j \le x < c_{j+1})$, an indicator that is 1 inside the bin and 0 outside. This is a piecewise constant fit, popular where natural breakpoints exist - five-year age bands, for instance - and poor where they do not, since a bin cannot show a trend inside itself.

The crucial word is fixed. The $b_j$ are chosen before fitting, not estimated, so the model is a linear model in the transformed columns. Least squares applies unchanged, and so does everything built on it: standard errors, $F$-statistics, confidence intervals. You have bought curvature without leaving the linear model.

Piecewise polynomials

Instead of one global polynomial, fit a separate low-degree polynomial in each region. Place knots $\xi_1, \dots, \xi_K$ through the range and fit $K + 1$ cubics, one per region. With one knot at $x = 50$:

$$f(x) = \begin{cases} \beta_{01} + \beta_{11}x + \beta_{21}x^2 + \beta_{31}x^3 & x < 50 \\ \beta_{02} + \beta_{12}x + \beta_{22}x^2 + \beta_{32}x^3 & x \ge 50 \end{cases}$$

Each piece is an ordinary cubic fit on its own subset. Two cubics, four parameters each: eight degrees of freedom.

What goes wrong

Nothing forces the two pieces to agree at the knot, so they do not. Fitting this on 200 simulated points with knots at 40, 50 and 60 - four cubics, sixteen parameters - the fitted value jumps at every knot, each jump taken before rounding:

$$\begin{array}{lll} \text{knot } 40 & 106.76 \to 100.43 & \text{jump } 6.34 \\ \text{knot } 50 & 94.74 \to 100.29 & \text{jump } 5.55 \\ \text{knot } 60 & 90.52 \to 96.78 & \text{jump } 6.26 \end{array}$$

These are not rounding artefacts. The curve is genuinely discontinuous, and as a description of a smooth relationship it is nonsense.

Constraints, and what they cost

The fix is not fewer knots but constraints. Require the pieces to meet:

1. Continuity at the knot. The V-shaped join this produces still looks wrong.
2. Continuous first derivative - no corner.
3. Continuous second derivative - no visible change in curvature.

Each constraint removes one free parameter. Starting from eight with one knot:

$$8 - 3 = 5 .$$

The result is a cubic spline. In general a cubic spline with $K$ knots uses

$$K + 4$$

degrees of freedom, because each additional knot adds a cubic (four parameters) and three constraints, a net cost of one.

The general definition. A degree-$d$ spline is a piecewise degree-$d$ polynomial with continuous derivatives up to order $d - 1$ at each knot. A linear spline is continuous with a corner allowed; the piecewise constant functions above are degree-0 splines, where even continuity is dropped.

Cubic is the usual choice for a simple reason: a discontinuity in the third derivative is not something the eye can detect.

Here is that accounting as a dial, run on the two hundred points above. At zero constraints the four cubics jump by the three numbers in the table; each constraint then removes one parameter per knot, walking the count 16, 13, 10, 7 and ending at the K + 4 of a cubic spline.

Two things the figure shows that the argument above leaves out. Those nine parameters buy almost nothing: the residual sum of squares rises by under three per cent across the whole journey, so the constraints are close to free. And the corner gets worse before it gets better. Imposing continuity alone more than doubles the sharpest change of slope, from 2.42 to 5.95, which is the V-shaped join mentioned above, measured: making the pieces touch is not the same as making them agree which way to go.

Where this is heading

We now have a fitting problem with constraints, which is less convenient than plain least squares. The next lesson removes that inconvenience entirely: one well-chosen extra column per knot enforces all three constraints automatically, turning the whole thing back into an ordinary regression.

Before the quiz

Be able to write polynomial and step regression as basis models, say why a basis model is still least squares, describe how a piecewise polynomial fails and by roughly how much, and count degrees of freedom before and after the constraints.