Estimators, Bias and Standard Error

You have a sample and you want to know something about the population it came from. You compute a number. The whole of this path rests on one shift in how you read that number, so it is worth stating plainly before anything else.

The number you computed is not the number you want

Suppose you want the average height of adults in a city, and you measure 40 people. Their average is $172.3$ cm. That figure is a fact about your 40 people. It is not a fact about the city.

Measure a different 40 and you would get something else - $171.1$, or $173.6$. The quantity you actually want, the population mean, sits somewhere fixed and unknown. What moves is your estimate of it.

An estimator is the recipe ("take the average of the sample"). An estimate is what the recipe returns on one particular sample. Because the sample is random, the estimate is a random variable, and it has a distribution - called the sampling distribution. Almost everything in inference is a statement about that distribution rather than about your one number.

Two features of it matter.

Bias asks whether the recipe is aimed correctly. If you repeated the study endlessly and averaged all the estimates, would you land on the truth? If yes, the estimator is unbiased.

Variance asks how tightly the estimates cluster. Two recipes can both be aimed correctly while one scatters far more widely than the other.

Aim and spread are separate questions, and neither alone tells you whether an estimate is any good.

Why dividing by n is wrong, precisely

The sample mean is unbiased for the population mean, which is unsurprising. The sample variance is more interesting, because the obvious recipe is wrong in a way that can be quantified exactly.

The obvious recipe is: take each observation's distance from the sample mean, square it, and average over the $n$ observations. It produces answers that are systematically too small - and not by a vague amount, but by the exact factor

$$\frac{n-1}{n}$$

At $n = 10$ that is nine tenths: on average you recover 90% of the true variance, no matter how many studies you run.

The reason is worth pausing on rather than memorising. You measured distances from the sample mean. But the sample mean is, by construction, the single point that makes those squared distances as small as they can possibly be for this sample. The true mean is somewhere else, so distances from it would have been larger. You have measured spread from an artificially convenient centre.

One degree of freedom was spent locating that centre, and dividing by $n-1$ instead of $n$ gives it back exactly. Simulating 400,000 samples of size 10 from a population with variance $4$:

Being right on average is not the same as being close

Here is where the story usually stops, and where it becomes interesting.

Unbiasedness is a statement about an average over infinitely many repetitions. You get one sample. What you actually care about is how far your estimate is likely to land from the truth, and the standard measure of that is mean squared error:

$$\text{MSE} = \mathbb{E}\big[(\hat\theta - \theta)^2\big] = \text{bias}^2 + \text{variance}$$

The decomposition is exact, and it means an estimator can trade one for the other. Accept a little bias, buy a larger reduction in variance, and end up closer to the truth on the whole.

That is not a hypothetical. For the two variance recipes at $n = 10$ with a true variance of $4$ and a normal population, the exact mean squared errors are

$$\text{divide by } n: \; \frac{\sigma^4(2n-1)}{n^2} = 3.04 \qquad \text{divide by } n-1: \; \frac{2\sigma^4}{n-1} = 3.555556$$

The biased estimator is closer to the truth. Dividing by the larger number shrinks every estimate slightly toward zero; the variance that shrinkage saves outweighs the bias it introduces.

This is not a curiosity to file away. It is the same trade that justifies ridge regression, shrinkage estimators and essentially all regularisation: a deliberate, quantified bias that lowers total error. Meeting it here, on a formula you already know, makes it much harder to mistake later for a compromise.

All four of those numbers have closed forms, so the figure below computes them rather than simulating: 3.6 and 4 for the two recipes on average, 3.04 and 3.5556 for their mean squared errors. Dragging the sample size settles something this section states at one size only - for normal data the biased recipe is closer at EVERY n, because (2n-1)(n-1) is less than 2n squared for every n, and the advantage shrinks away as the sample grows.

The standard error, and what it costs

The standard deviation of the sampling distribution has its own name - the standard error - because it is what "how precise is this estimate?" means. For a sample mean it is

$$\text{SE}(\bar{X}) = \frac{\sigma}{\sqrt{n}}$$

The square root governs the economics of every study ever designed. With a population standard deviation of $2$:

• at $n = 25$, the standard error is $0.4$
• at $n = 100$, it is $0.2$

Halving your uncertainty cost you four times the data. Going from $0.2$ to $0.1$ would cost 300 observations more. Precision is purchasable, but the price rises steeply, which is why underpowered studies cannot be rescued by adding a few more subjects - a point the third lesson makes quantitative.

What this buys you

Three ideas, and the rest of the path is built on them.

An estimate is a draw from a distribution, so the honest question is never "what is the answer?" but "how does this recipe behave across the samples I might have drawn?"

Error splits into aim and spread, and the split is exact. Improving an estimator usually means trading between them rather than eliminating either.

Precision improves with $\sqrt{n}$, so it is bought at a rising price and must be budgeted before data collection rather than hoped for afterwards.

The next lesson attaches a range to an estimate, and finds that the standard recipe for doing so promises more than it delivers.