Logistic Regression and Classification

Linear Regression from First Principles fitted a numeric response. Many problems instead ask a categorical question: does this belong to class A or class B? What we usually want is not the label but the probability of the label, and probabilities are constrained in a way that straight lines are not.

A. Why not just fit a line?

Code the two classes as $0$ and $1$ and run least squares. The fitted line is $\hat\beta_0 + \hat\beta_1 X$, defined for every real $X$, and therefore eventually exceeds $1$ and eventually falls below $0$. As an estimate of $P(Y = 1 \mid X)$ that is not merely inaccurate, it is impossible - it violates the axioms from Probability from Zero.

We need a model whose output is confined to $(0, 1)$ by construction.

B. The logistic function

Logistic regression models the probability directly:

$$p(X) = P(Y = 1 \mid X) = \frac{e^{\beta_0 + \beta_1 X}}{1 + e^{\beta_0 + \beta_1 X}} .$$

Writing $z = \beta_0 + \beta_1 X$, this is the logistic (or sigmoid) function

$$\sigma(z) = \frac{1}{1 + e^{-z}} ,$$

the two forms being algebraically identical (divide numerator and denominator of the first by $e^{z}$). Its behaviour is exactly what we need: as $z \to +\infty$, $\sigma \to 1$; as $z \to -\infty$, $\sigma \to 0$; and $\sigma(0) = 0.5$. The output can approach the endpoints but never reach them.

A few values, with $z = -4 + x$:

C. Odds and log-odds: what the coefficients mean

Rearranging the model gives the odds:

$$\frac{p(X)}{1 - p(X)} = e^{\beta_0 + \beta_1 X} ,$$

and taking logs gives the log-odds or logit:

$$\log\!\left(\frac{p(X)}{1 - p(X)}\right) = \beta_0 + \beta_1 X .$$

This is the sense in which logistic regression is a linear model - linear in the log-odds, not in the probability. Odds run from $0$ to $\infty$ and log-odds over the whole real line, which is precisely the range a linear function needs.

So $\beta_1$ is the change in log-odds per one-unit increase in $X$, and $e^{\beta_1}$ is the multiplicative change in the odds.

The most common misreading. $\beta_1$ is not the change in probability per unit of $X$. Because the logistic curve is S-shaped, the same one-unit step moves the probability a lot near $p = 0.5$ and almost not at all out in the tails. Only the log-odds change by a constant amount.

D. Fitting by maximum likelihood

Least squares is not the natural criterion here. Instead we choose the coefficients that make the observed labels most probable. Each observation contributes $p(x_i)$ if $y_i = 1$ and $1 - p(x_i)$ if $y_i = 0$, which combines into the likelihood

$$\ell(\beta_0, \beta_1) = \prod_{i:\,y_i=1} p(x_i) \prod_{i:\,y_i=0}\big(1 - p(x_i)\big) .$$

Taking logs turns the product into a sum - numerically far better behaved - giving the log-likelihood

$$\log \ell = \sum_{i=1}^{n}\Big[y_i \log p(x_i) + (1-y_i)\log\big(1 - p(x_i)\big)\Big] .$$

Unlike least squares, this has no closed-form solution; it is maximised numerically. Its gradient is remarkably clean:

$$\frac{\partial \log\ell}{\partial \beta_0} = \sum_i (y_i - p_i), \qquad \frac{\partial \log\ell}{\partial \beta_1} = \sum_i (y_i - p_i)\,x_i .$$

Each observation pushes the coefficients in proportion to how wrong the current prediction is - a pattern that reappears in Backpropagation and Gradient Descent.

E. One gradient step, by hand

Six observations, deliberately not perfectly separable (the reason is in section G):

Start from $\beta_0 = \beta_1 = 0$. Then $z_i = 0$ and $p_i = \sigma(0) = 0.5$ for every observation, so

$$\log\ell = 6\log(0.5) = -4.158883 .$$

The gradient components:

$$\sum_i (y_i - p_i) = (0-0.5)+(1-0.5)+(0-0.5)+(1-0.5)+(1-0.5)+(1-0.5) = 1.0 ,$$ $$\sum_i (y_i - p_i)x_i = -0.5 + 1.0 - 1.5 + 2.0 + 2.5 + 3.0 = 6.5 .$$

Taking a step of size $\eta = 0.05$ uphill (we are maximising):

$$\beta_0 \leftarrow 0 + 0.05(1.0) = 0.05, \qquad \beta_1 \leftarrow 0 + 0.05(6.5) = 0.325 .$$

The new probabilities are $[0.5927, 0.6682, 0.7359, 0.7941, 0.8422, 0.8808]$ and the log-likelihood has risen to $-3.162034$. One step improved the fit.

Step size is not a free parameter. With $\eta = 0.5$ the same step lands at $\beta = (0.5, 3.25)$ and the log-likelihood falls to $-14.024$ - far worse than where it started. Overshooting a maximum is easy; that a step is uphill locally does not mean any step size along it is an improvement.

Iterating to convergence gives

$$\hat\beta_0 = -2.7700, \qquad \hat\beta_1 = 1.1447,$$

with log-likelihood $-2.440125$.

Interpreting the result. Each extra unit of $x$ multiplies the odds by $e^{1.14466} = 3.1414$ - roughly tripling them. The decision boundary, where $p = 0.5$, is where $z = 0$:

$$x = -\frac{\hat\beta_0}{\hat\beta_1} = \frac{2.7700}{1.14466} = 2.4199 .$$

Below $x \approx 2.42$ we predict class 0, above it class 1.

Every number above is reproduced by running this.

F. From probability to decision

The model outputs a probability; a decision needs a threshold. The default $0.5$ is only correct when the two kinds of mistake cost the same. When a false negative is far more costly than a false positive, the right threshold is lower - and this is a decision about consequences, not a statistical question the data can answer.

The base-rate lesson from Bayes' Theorem applies here in full: with a rare positive class, a model can be highly accurate and still be wrong most of the times it predicts "positive".

G. When the fit blows up

If the classes are perfectly separable - some threshold splits them with no overlap - the likelihood can always be increased by making $\beta_1$ larger, pushing fitted probabilities towards exactly $0$ and $1$. There is no maximum, and the MLE does not exist.

This is not hypothetical. Replacing our data with the cleanly separated $x = (1,2,3,4)$, $y = (0,0,1,1)$ makes an unregularised solver return $\hat\beta_1 \approx 18.2$ with $\hat\beta_0 \approx -45.8$ - numbers that are artefacts of where the optimiser stopped, not estimates. Software may or may not warn you. The standard remedy is a penalty on coefficient size, which is exactly the subject of the next article.

Key takeaways

• A linear model of a probability produces values outside $[0,1]$; the logistic function confines the output to $(0,1)$ by construction.
• Logistic regression is linear in the log-odds: $\beta_1$ is the change in log-odds per unit of $X$, and $e^{\beta_1}$ the odds multiplier - not a change in probability.
• Fitting maximises the log-likelihood, which has no closed form and is solved numerically; the gradient is $\sum(y_i - p_i)$ and $\sum(y_i - p_i)x_i$.
• Our fit converged to $\hat\beta_0 = -2.7700$, $\hat\beta_1 = 1.1447$: odds $\times 3.14$ per unit, boundary at $x = 2.4199$.
• Step size matters - $\eta = 0.05$ improved the fit, $\eta = 0.5$ made it much worse.
• Under perfect separation the MLE does not exist and coefficients diverge.

What's next

Both failure modes just seen - coefficients running away under separation, and flexible models chasing noise - are treated by the same idea: add a penalty that makes large coefficients expensive. That is Regularization: Ridge and Lasso.