Backpropagation and Gradient Descent

A neural network is a chain of simple operations, each with adjustable weights. Training means finding weights that make the output right. Two ideas do all the work: gradient descent, which says which way to move the weights, and backpropagation, which computes the gradient efficiently no matter how deep the chain is.

Backpropagation is often described as difficult. It is the chain rule, applied in a specific order. This article computes every derivative of a small network by hand and then checks the results against an automatic differentiation engine.

A. Layers, weights, and the loss

A dense layer applies a linear transformation followed by an element-wise non-linearity:

$$\mathbf{a} = g(W\mathbf{x} + \mathbf{b}) ,$$

where $W$ is a weight matrix, $\mathbf{b}$ a bias vector, and $g$ an activation function. Without $g$, stacking layers would be pointless: a composition of linear maps is just another linear map, and the network could represent nothing a single layer could not.

We use the ReLU, $g(z) = \max(0, z)$, whose derivative is $1$ for $z > 0$ and $0$ for $z < 0$.

A loss function scores the output against the target. For a scalar output we use

$$L = \tfrac{1}{2}(\hat y - y)^2 ,$$

the $\tfrac12$ being a convenience that cancels when differentiated.

The key reframing: with the data held fixed, $L$ is a function of the weights. Training is minimising that function.

B. Gradient descent

The gradient $\nabla L$ points in the direction of steepest increase, so to decrease $L$ we step against it:

$$w \leftarrow w - \eta \frac{\partial L}{\partial w} ,$$

with $\eta$ the learning rate. Too small and training crawls; too large and it overshoots - the same failure demonstrated numerically in Logistic Regression, where a step of $0.5$ made the objective substantially worse.

So the entire problem reduces to computing $\partial L / \partial w$ for every weight. A network can have millions, and computing each independently would be hopeless. Backpropagation gets all of them in a single backward pass.

C. The network we will differentiate

Two inputs, a two-unit ReLU hidden layer, one linear output. Biases are zero.

$$\mathbf{x} = \begin{bmatrix}1\\2\end{bmatrix},\quad W^{(1)} = \begin{bmatrix}0.1 & 0.3\\ 0.2 & 0.4\end{bmatrix},\quad W^{(2)} = \begin{bmatrix}0.5 & -0.5\end{bmatrix},\quad y = 1 .$$

Forward pass.

$$\mathbf{z}^{(1)} = W^{(1)}\mathbf{x} = \begin{bmatrix}0.1(1) + 0.3(2)\\ 0.2(1) + 0.4(2)\end{bmatrix} = \begin{bmatrix}0.7\\ 1.0\end{bmatrix} .$$

Both entries are positive, so ReLU passes them through unchanged:

$$\mathbf{a}^{(1)} = \begin{bmatrix}0.7\\ 1.0\end{bmatrix} .$$ $$\hat y = z^{(2)} = 0.5(0.7) + (-0.5)(1.0) = 0.35 - 0.5 = -0.15 .$$ $$L = \tfrac12(-0.15 - 1)^2 = \tfrac12(-1.15)^2 = \tfrac12(1.3225) = 0.66125 .$$

D. The backward pass

We now walk backwards, carrying the derivative of $L$ with respect to each quantity as we go. Each step is one application of the chain rule.

Step 1 - the output.

$$\frac{\partial L}{\partial \hat y} = \hat y - y = -0.15 - 1 = -1.15 .$$

Step 2 - the output weights. Since $\hat y = W^{(2)}\mathbf{a}^{(1)}$, we have $\partial \hat y / \partial W^{(2)}_j = a^{(1)}_j$, so

$$\frac{\partial L}{\partial W^{(2)}} = \frac{\partial L}{\partial \hat y}\,\mathbf{a}^{(1)} = -1.15 \begin{bmatrix}0.7 & 1.0\end{bmatrix} = \begin{bmatrix}-0.805 & -1.15\end{bmatrix} .$$

Note the structure: the gradient of a weight is the error signal arriving at it multiplied by the activation flowing into it. That pattern holds at every layer.

Step 3 - back through the output layer. To continue we need how $L$ changes with the hidden activations:

$$\frac{\partial L}{\partial \mathbf{a}^{(1)}} = \frac{\partial L}{\partial \hat y}\,W^{(2)} = -1.15\begin{bmatrix}0.5 & -0.5\end{bmatrix} = \begin{bmatrix}-0.575 & 0.575\end{bmatrix} .$$

The error is distributed backwards in proportion to the weights that carried the signal forward.

Step 4 - through the ReLU. Multiply element-wise by the activation's derivative. Both pre-activations were positive, so both derivatives are $1$:

$$\frac{\partial L}{\partial \mathbf{z}^{(1)}} = \begin{bmatrix}-0.575 & 0.575\end{bmatrix} .$$

This is where dead units come from. Had a pre-activation been negative, the ReLU derivative would be $0$ and the gradient would be annihilated - no error signal reaches any weight feeding that unit, and it cannot learn. A unit stuck negative for all inputs is a dead unit, and it is the standard motivation for variants such as leaky ReLU.

Step 5 - the input weights. Same pattern as step 2: error signal times incoming activation, here the input itself.

$$\frac{\partial L}{\partial W^{(1)}} = \frac{\partial L}{\partial \mathbf{z}^{(1)}} \mathbf{x}^{\top} = \begin{bmatrix}-0.575\\ 0.575\end{bmatrix}\begin{bmatrix}1 & 2\end{bmatrix} = \begin{bmatrix}-0.575 & -1.15\\ 0.575 & 1.15\end{bmatrix} .$$

Every gradient in the network, from one forward pass and one backward pass.

E. Checking against autograd

Hand derivations are error-prone, and there is no reason to trust one that has not been checked:

Both print True: the hand-computed gradients agree with PyTorch's autograd to floating-point precision.

F. Taking the step

Every weight moves against its own gradient. With $\eta = 0.1$:

$$W^{(2)} \leftarrow \begin{bmatrix}0.5 & -0.5\end{bmatrix} - 0.1\begin{bmatrix}-0.805 & -1.15\end{bmatrix} = \begin{bmatrix}0.5805 & -0.385\end{bmatrix} ,$$ $$W^{(1)} \leftarrow \begin{bmatrix}0.1 & 0.3\\ 0.2 & 0.4\end{bmatrix} - 0.1\begin{bmatrix}-0.575 & -1.15\\ 0.575 & 1.15\end{bmatrix} = \begin{bmatrix}0.1575 & 0.415\\ 0.1425 & 0.285\end{bmatrix} .$$

Recomputing the forward pass with both updated matrices gives $\hat y = 0.2989$ and

$$L = \tfrac12(0.2989 - 1)^2 = 0.2457 ,$$

down from $0.66125$. One step cut the loss by more than half. Note that $\hat y$ moved from $-0.15$ towards the target $y = 1$ - it overshot none of the way, it simply moved in the right direction, which is all a gradient step promises.

The figure below is this network with every number on it: the activations on the units, and on each weight the gradient that reaches it. Check any of them against the arithmetic above. Then move the learning rate. The lesson is right that the steps shrink on their own near a minimum, and it is worth knowing how close the cliff is on the other side: at 0.5, five times the rate used here, the first oversized step drives one hidden unit negative and the third drives the other, ReLU then blocks every gradient beneath them, and the network sits at a loss of 0.5 for ever. That is not divergence. It is death, and it takes three steps.

G. Batches, and why the algorithm scales

Real training does not use one example. Stochastic gradient descent computes the gradient on a small random mini-batch and steps, repeating over the dataset. The batch gradient is noisier than the full-dataset gradient but vastly cheaper, and the noise is often helpful - it can knock the parameters out of poor regions.

Backpropagation's efficiency is what makes any of this feasible: one forward pass and one backward pass yield the derivative with respect to every parameter, with cost proportional to the forward pass rather than to the number of parameters. Computing each of a million partial derivatives separately by finite differences would need a million forward passes.

The gradient is local, and the loss surface is not convex. Unlike least squares, a network's loss has many minima and saddle points, and gradient descent offers no guarantee of finding the best one. In practice good-enough minima are common, but "it converged" is not "it found the optimum".

Key takeaways

• Training reframes the loss as a function of the weights and minimises it by gradient descent, $w \leftarrow w - \eta\,\partial L/\partial w$.
• Non-linear activations are essential; stacked linear layers collapse to one.
• Backpropagation is the chain rule applied backwards, reusing each intermediate derivative.
• Every weight gradient is incoming activation × outgoing error signal - the same rule at every layer.
• Our worked network: $L = 0.66125$, $\partial L/\partial W^{(2)} = [-0.805, -1.15]$, $\partial L/\partial W^{(1)} = [[-0.575, -1.15], [0.575, 1.15]]$, confirmed against autograd.
• ReLU units with negative pre-activations pass zero gradient and can die.

What's next

The same backward pass trains the architecture behind modern language models, but those models need a mechanism for letting each position in a sequence consult every other. That mechanism is Attention and Self-Attention.