3 articles.
The step size you are allowed is fixed by the steepest direction and the number of steps you need is fixed by the flattest, so the cost of gradient descent is their ratio. The same least-squares fit, to the same ten decimal places, takes 1742 steps in one basis, 147 in a rescaled one and exactly 1 in an orthonormal one, and momentum buys back the square root of the ratio rather than the ratio.
Allowing sideways moves takes hill climbing on 8 queens from 14.75% of runs solved to 94.55%, which reads like a six-fold improvement and is not one: with random restarts the expected cost of a solution goes from 21.9 steps to 23.1, and counted in moves evaluated it falls by 16%, from 1,547 to 1,298. Simulated annealing solves 98.8% and costs 1,622 evaluations. What changed was mostly the statistic, not the work.
A two per cent change in the learning rate separates a converged run from one five orders of magnitude away, a condition number predicts the convergence rate to six decimal places, and stochastic gradient descent with a fixed step never converges at all - it settles into a ball whose radius grows as the square root of the step. Every figure here was computed on a problem whose exact optimum is known.