Principal Components and Variance

Six points, two measurements each. By the end of this lesson one number will replace both of them and keep 98.83% of everything that varied.

That is the whole idea of principal components: most datasets with many columns are not really that wide. The columns move together, and the movement lies close to a lower-dimensional shape. Finding that shape lets you carry almost all of the information in far fewer numbers.

The catch is that there is nothing to check your answer against. Everything in the supervised path had a response to predict, so a model could be right or wrong. Here there is no response at all - only the data - and "what structure does it contain" is a genuinely harder question even to pose. So the first job is to say precisely what a good answer would mean.

Principal component analysis makes the question precise in one way: find a small number of directions along which the data vary as much as possible, and use them in place of the original variables.

The first principal component

The first principal component is the normalised linear combination

$$Z_1 = \phi_{11}X_1 + \phi_{21}X_2 + \dots + \phi_{p1}X_p, \qquad \sum_{j=1}^{p}\phi_{j1}^2 = 1,$$

that has the largest variance. The constraint is not decoration. Without it you could double every $\phi_{j1}$, quadruple the variance, and repeat forever - the maximisation would have no solution at all. Fixing the length to one makes the problem about direction rather than magnitude.

The vector $\phi_1 = (\phi_{11}, \dots, \phi_{p1})$ is the loading vector. It describes a direction in feature space, and it belongs to the dataset as a whole. Projecting the observations onto it gives the scores $z_{11}, \dots, z_{n1}$, one per observation.

Loadings and scores answer different questions. A loading says how much a variable contributes to a component; a score says where an observation sits along it. Confusing the two is the most common way to misread a PCA output.

James et al. add a second, equivalent characterisation worth keeping: the first component direction is also the line closest to all $n$ observations, in the sense of squared perpendicular distance. Maximising the variance of the projections and minimising the distance to the line are the same problem.

Worked example

Six observations on two variables:

$$X = \begin{bmatrix} 2&0\\ 3&1\\ 4&3\\ 5&4\\ 6&7\\ 4&2 \end{bmatrix}, \qquad \bar x = (4.0000,\; 2.8333).$$

Centre the columns and the sample covariance matrix is

$$S = \begin{bmatrix} 2.0000 & 3.4000 \\ 3.4000 & 6.1667 \end{bmatrix}.$$

Its eigenvalues are $\lambda_1 = 8.0708$ and $\lambda_2 = 0.0958$, and the first eigenvector - the first loading vector - is

$$\phi_1 = (0.4886,\; 0.8725), \qquad \|\phi_1\| = 1 .$$

Both loadings are positive and the second is larger, so the component is essentially "both variables together, weighted toward the second". The scores are

$$z_{i1} = (-3.4493,\; -2.0882,\; 0.1454,\; 1.5065,\; 4.6126,\; -0.7271),$$

and their sample variance is $8.0708$ - exactly $\lambda_1$. That is not a coincidence: the eigenvalue is the variance captured by its component.

Proportion of variance explained

Each component captures some share of the total variance, and the proportion of variance explained (PVE) reports it:

$$\text{PVE}_m = \frac{\lambda_m}{\sum_{k=1}^{p}\lambda_k}.$$

Here $\text{PVE}_1 = 8.0708 / 8.1667 = 0.9883$ and $\text{PVE}_2 = 0.0117$. One direction carries 98.83% of the variation in a two-variable dataset, so replacing both variables with a single score loses barely anything.

A scree plot puts the PVE of each component in order, and the usual advice is to look for an elbow. It is worth being honest about what that is: an eyeball judgement, not a test. There is no widely accepted objective way to decide how many components are enough, and in an unsupervised setting there is no held-out error to appeal to either - which is exactly the difficulty of having no $y$.

Why standardising is not optional

PCA maximises variance, and variance has units. Measure a length in millimetres rather than metres and its variance grows by a factor of $10^6$, so that variable will dominate the first component for no reason but the choice of ruler.

The fix is to scale each variable to standard deviation one before computing the components - which amounts to running PCA on the correlation matrix rather than the covariance matrix. James et al. describe this as the usual practice, and the exception is worth naming: when the variables are already in the same units and their differing variances are genuinely meaningful, scaling would throw away real information.

Before the quiz

Be able to state what the first component maximises and why the norm constraint is needed, distinguish a loading from a score, compute a PVE from eigenvalues, and say what goes wrong if you skip standardisation. The encyclopedia entry Principal Component Analysis covers the same material as a reference.