What Actually Makes Training Converge

A learning rate of 1.1246 trains a model. A learning rate of 1.1473 sends the same model, on the same data, from the same starting point, to a loss five orders of magnitude worse.

Nothing about the model changed. The gap between those two numbers is two per cent, and there is no gradual degradation in between - the run either settles or is flung away. That cliff has an exact location, it can be computed before training starts, and the same small piece of theory that locates it also predicts how many steps a successful run will take.

Everything below is measured on one deliberately small problem: least squares with 400 points and two parameters, whose exact optimum $w^\star$ is known in closed form. Knowing the answer in advance is the whole point. It turns every claim into a measurement rather than an impression.

A. The cliff

Near a minimum the loss is a quadratic bowl, and the bowl's shape is the Hessian $H$. Write the error as $e_t = w_t - w^\star$ and one gradient step is just

$$e_{t+1} = (I - \eta H)\, e_t .$$

Decompose the error along the eigenvectors of $H$. Each component is multiplied by $1 - \eta\lambda$, where $\lambda$ is that direction's curvature, and the directions never interact. A single step is therefore not one motion - it is as many independent contractions as the problem has curvature directions, each running at its own rate.

A component shrinks only when $\lvert 1 - \eta\lambda\rvert < 1$, which is exactly $\eta < 2/\lambda$. Since every direction has to shrink, the binding constraint is the sharpest curvature $L$:

$$\eta < \frac{2}{L}.$$

On this problem $L = 1.760627$, so the threshold is $2/L = 1.135959$. The two step sizes at the top of this article are 0.99 and 1.01 times that number, and after 200 steps they leave the loss at 0.236592 and at 23585.65.

That is the cliff. It is not a region of instability with a grey zone; it is a sign change in $1 - \eta L$, after which the sharpest direction is multiplied by the same factor greater than one every single step. Anyone who has watched a loss curve go to NaN within thirty steps of a "small" learning-rate increase has seen this exact arithmetic.

B. Stable is not fast

Knowing $L$ does not mean setting $\eta = 2/L$, because stability and speed are different questions with different answers.

The error contracts by $\max(\lvert 1-\eta L\rvert, \lvert 1-\eta\mu\rvert)$ per step, where $\mu$ is the flattest curvature. Raising $\eta$ helps the flat direction and hurts the sharp one, so the best step is where the two costs meet, at $\eta^\star = 2/(L+\mu) = 1.090230$. Every step size between that and the threshold is both slower and closer to the cliff.

It is worth checking such a derivation rather than trusting it. Running the same 200 steps at each of 2000 step sizes and keeping the one that ends closest to $w^\star$ gives 1.088582 against the derived 1.090230 - agreement to $1.6\times10^{-3}$. The small gap is not the grid: over a finite 200 steps the best step sits just below the asymptotic optimum, and closes on it as the run lengthens. A formula with a sign slip in it still looks like a formula; a grid search costs three lines and does not.

C. One number predicts the whole run

Put $\eta^\star$ back in and the contraction becomes a function of a single quantity - the condition number $\kappa = L/\mu$, the elongation of the bowl:

$$\rho_{\mathrm{GD}} = \frac{\kappa - 1}{\kappa + 1}.$$

Here $\kappa = 23.8410$, so $\rho_{\mathrm{GD}} = 0.919488$. Measuring the real run - taking $\lVert w_t - w^\star\rVert$ at steps 20 and 100 and extracting the per-step ratio - gives $0.919488$. Six decimal places, with nothing fitted: the prediction uses two eigenvalues of a matrix and reproduces a run that knows nothing about them.

Note how little the rate depends on. Not the data beyond its Hessian, not the starting point, not the dimension. And note the scale: $\kappa \approx 24$ is a mild condition number, and it already costs 165 steps to shrink the error a millionfold on a two-parameter problem. Real networks run in the thousands.

This is also the reason rescaling a single input column can transform a training time. $\kappa$ is a property of the parameterisation, not of what is being learned. Standardising features, batch normalisation and layer normalisation all move the Hessian's eigenvalues, and moving them is moving the rate. They are not statistical politeness; they are conditioning.

D. What momentum is actually doing

Gradient descent throws away the history of the run at every step - including the fact that the gradient along the flat direction has pointed the same way for a hundred steps, while the gradient along the sharp direction has been alternating in sign.

Momentum keeps a running velocity, $v_{t+1} = \beta v_t - \eta\nabla f(w_t)$, and the two directions get opposite treatment from it for free. Down the valley, where successive gradients agree, the decaying average builds to roughly $1/(1-\beta)$ times a single gradient. Across the valley, where they alternate, they largely cancel. One blind mechanism amplifies exactly the slow direction and damps exactly the oscillating one, and the rate improves to

$$\rho_{\mathrm{mom}} = \frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1} = 0.660022 .$$

The dependence on the condition number went from $\kappa$ to $\sqrt{\kappa}$. Momentum reaches millionfold accuracy in 42 steps against 165 - a factor of 3.9, where the asymptotic rates predict $164.6/33.3 = 4.9$.

The shortfall is worth keeping rather than rounding away. Polyak's rate is asymptotic. With his tuning both curvature directions sit exactly at the extremes, so each gets a repeated root, at $-0.660022$ and $+0.660022$, and a repeated root decays like $t\rho^t$ rather than $\rho^t$. Measured over steps 20 to 60 the real contraction is 0.677786, close to the $3^{1/40} \times 0.660022 = 0.6784$ that the extra factor of $t$ predicts. On this problem the asymptotics never arrive, because the run hits machine precision first. So the honest claim is not that momentum converges at 0.660022, but that it turns a $\kappa$ problem into a $\sqrt\kappa$ problem and you should expect most of that on a real run.

Both runs are below, each at its own optimal settings, with the error on a log axis. At the condition number this problem actually has, the counts are the ones just quoted: 165 steps against 42. The slider is the point, though. A change of exponent is the one thing a pair of numbers cannot show, so drag the condition number and watch the two curves separate - the plain count growing with kappa and the momentum count with its square root. The two rate read-outs are also kept apart deliberately: the asymptotic one is a limit, and the measured one is what this run actually does.

E. And then the gradient is not exact

Everything so far assumed the true gradient. Real training has a minibatch estimate, and that single substitution breaks the guarantee that any of this converges.

The estimate is unbiased - a uniform sample of per-example gradients has the full gradient as its expectation - so a small batch does not point somewhere systematically different. What changes is spread: at the origin $w = (0, 0)$, where the full gradient is $(-3.416495, 0.179597)$, the typical noise is 1.4382 at $B = 8$ and 0.3027 at $B = 128$. Sixteen times the compute for 4.75 times the precision, close to the $\sqrt{16}$ that averaging predicts and a little above it because these batches are drawn without replacement from only $n = 400$ points, which predicts $4\sqrt{392/272} = 4.80$. Linear cost, square-root benefit: that exchange rate is the entire argument for small batches and many steps.

The consequence is that a fixed-step run never converges. Each step contracts toward $w^\star$ and each step injects fresh sampling noise, and at some radius the two balance. Running to 120,000 steps and measuring the root-mean-square distance over the tail:

None of these is heading to zero. And the exchange rate is worse than it looks: halving $\eta$ from 0.20 to 0.10 shrinks the floor by 31%, not 50%. Fitting across the whole sixteenfold range gives an exponent of 0.5090 - the radius scales as $\sqrt\eta$, because the stationary variance is $O(\eta)$ and distance is its square root.

Sixteen times the patience for four times the accuracy is a bad enough bargain that the constant step is not tuned down but abandoned. Replacing it with $\eta_t = 0.20/(1+t/500)$ - same initial rate, same data, same seed - brings the run to 0.009909, eleven times closer over the same horizon and still improving when stopped.

What makes a schedule work is a pair of conditions that pull against each other: $\sum_t \eta_t = \infty$, so the run keeps enough total travel to arrive from anywhere, and $\sum_t \eta_t^2 < \infty$, so the injected noise is summable and the ball can close. A constant step meets the first and fails the second. A step decaying like $1/t^2$ fails the first and can run out of travel before arriving.

None of that has to be simulated. On a quadratic the second moment of the error obeys an exact recursion, so the figure below runs the whole experiment as arithmetic: no seed, no 120,000 steps, and the floor available in closed form as eta sigma^2 / (lambda (2 - eta lambda)). Its noise scale is calibrated on the 0.2 row above and the other four are then predictions, landing within 2.5%. The closed form also explains the exponent: it is above a half because the contraction per step is 2 - eta lambda rather than 2.

F. The comparison that reverses

Which leaves the practical question: which optimizer? Same problem, same batches, same seed, each at a sensible fixed step - SGD at $\eta = 0.05$, momentum at $\eta = 0.01$ with $\beta = 0.9$, Adam at $\eta = 0.05$ - with $f^\star = 0.233943$:

At 200 steps the ranking is the expected one, and the gap is large. By 5000 steps it has reversed: plain SGD is closest to the optimum and the two faster methods have settled further out. One step of one run is noisy - with other batch seeds SGD leads at step 5000 only about half the time - but averaged over steps 3000 to 8000 the order SGD, momentum, Adam held in every one of 40 other seeds.

Nothing failed. Momentum and Adam each move further per step, which is the same thing as a larger effective $\eta$ - and by the $\sqrt\eta$ result, a larger effective step means a wider ball. The early leader finishes last precisely because of what made it fast.

Adam still earns its place on the coordinate it was designed for. Dividing by a running estimate of each coordinate's gradient scale is conditioning applied one coordinate at a time, and on the flat direction of this problem Adam is 0.018492 from the optimum after 1000 steps where SGD is still 0.178858 away.

The short version

Three results, one recipe. The step size has an exact ceiling set by the sharpest curvature, and a different, lower optimum set by the ratio of the extreme curvatures. Momentum and adaptive methods buy the early progress by attacking that ratio. A decaying schedule buys the final digits by closing the noise ball. Neither substitutes for the other, which is why every serious training recipe specifies both.

And when a run is stuck at a loss that will not move, the first question is not which optimizer to switch to. It is whether the run is at a minimum at all, or merely at the edge of a noise ball whose radius you can compute.