Least Squares from the Derivative

Linear regression is usually presented as a formula to apply. It is better understood as the answer to a calculus question: which line makes the squared errors smallest? Once you ask it that way, the coefficients fall out of two derivatives and there is nothing left to memorise.

The objective

Fit $\hat y_i = \beta_0 + \beta_1 x_i$ and measure the misfit by the residual sum of squares:

$$\mathrm{RSS}(\beta_0, \beta_1) = \sum_{i=1}^{n}\big(y_i - \beta_0 - \beta_1 x_i\big)^2 .$$

Squaring does two jobs: it stops positive and negative errors cancelling, and it makes RSS a smooth convex function, so setting derivatives to zero finds the global minimum rather than merely a stationary point.

Differentiating

With respect to the intercept:

$$\frac{\partial \mathrm{RSS}}{\partial \beta_0} = -2\sum_{i=1}^{n}\big(y_i - \beta_0 - \beta_1 x_i\big) = 0 \quad\Longrightarrow\quad \sum_{i=1}^{n} e_i = 0 ,$$

writing $e_i$ for the residual. The intercept condition guarantees the residuals sum to exactly zero - equivalently, that the fitted line passes through the point of means $(\bar x, \bar y)$. That is not a coincidence or a convention; it is what that one derivative says.

With respect to the slope, and substituting $\beta_0 = \bar y - \beta_1 \bar x$:

$$\hat\beta_1 = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sum_i (x_i - \bar x)^2}, \qquad \hat\beta_0 = \bar y - \hat\beta_1 \bar x .$$

Worked example

Five points: $x = (1,2,3,4,5)$, $y = (2.1, 3.9, 6.2, 7.8, 10.1)$.

$\bar x = 3.0$, $\bar y = 6.02$. Then

$$S_{xy} = \sum (x_i - \bar x)(y_i - \bar y) = 19.90 , \qquad S_{xx} = \sum (x_i - \bar x)^2 = 10.00 ,$$

so

$$\hat\beta_1 = \frac{19.90}{10.00} = 1.9900 , \qquad \hat\beta_0 = 6.02 - 1.99 \times 3 = 0.0500 .$$

The residuals are $(0.06,\, -0.13,\, 0.18,\, -0.21,\, 0.10)$ - and they sum to exactly $0$, as the intercept derivative promised. Check it yourself; that sum is the fastest way to catch an arithmetic slip.

R-squared, and its trap

$$R^2 = 1 - \frac{\mathrm{RSS}}{\mathrm{TSS}}, \qquad \mathrm{TSS} = \sum_i (y_i - \bar y)^2 .$$

Here $\mathrm{TSS} = 39.7080$ and $\mathrm{RSS} = 0.1070$, giving $R^2 = 0.9973$ - the line explains 99.73% of the variation in $y$.

Why $R^2$ cannot rank models of different size. Adding any predictor, even pure noise, gives least squares strictly more freedom to reduce RSS. RSS can therefore never increase when a predictor is added, so $R^2$ can never decrease. A metric that always rewards more predictors cannot be used to decide whether a predictor is worth having. Comparisons need something that penalises size - adjusted $R^2$, an information criterion, or a held-out estimate from cross-validation.

See the squares. The figure below uses a different five points, $(1,2), (2,4), (3,5), (4,4), (5,5)$, which scatter far more than the worked example. Each residual is drawn with its square, so RSS is their total area, set against the TSS that the flat mean leaves. Move Intercept and Slope, or press Snap to the least-squares line: it lands on $\hat y = 2.2 + 0.6x$, where $\mathrm{RSS} = 2.4$, $\mathrm{TSS} = 6$ and $R^2 = 0.6$, and the residuals again sum to exactly $0$.

See it move. Least squares is one member of a larger family: slide the parameter and watch a log-likelihood peak exactly where the estimator lands.

Try it live. Fit a generalised linear model with the same normal-equation step iterated to convergence:

Before the quiz

Be able to say what setting $\partial \mathrm{RSS}/\partial \beta_0 = 0$ guarantees, and why $R^2$ is the wrong tool for model selection. Full derivation: Linear Regression from First Principles.