The Treatment That Helps Everyone and Harms the Average

Here is a set of numbers. All of them are exact, none is a trick, and together they say something that sounds impossible.

A treatment is given to 60 patients and withheld from 60 others. Among the severe cases it raises recovery from $0.350$ to $0.400$. Among the mild cases it raises recovery from $0.650$ to $0.700$. Five points, in both groups.

Pooled across everyone, the treated recover $0.500$ and the untreated recover $0.550$. The treatment appears to harm.

Every patient belongs to exactly one of those two groups. The treatment helped in both. And it lowered the overall rate by exactly the same five points it raised each group by.

Where the numbers come from

Check the addition: $16 + 14 = 30$ out of $40 + 20 = 60$, and $7 + 26 = 33$ out of $20 + 40 = 60$. Both arms have 60 patients, so the pooled comparison is not being distorted by unequal sizes.

What differs is who is in each arm. Two thirds of the treated were severe; only one third of the untreated were. And severe patients recover less often whatever you do to them.

So the pooled figure is not comparing the treatment. It is comparing a group that is mostly severe against a group that is mostly mild, and the case mix overwhelms the five points the treatment contributes.

The mechanism has a name and a shape

Severity influenced who got treated - doctors gave the drug to the patients who needed it - and it influences who recovers. A variable with both of those arrows is a confounder.

Both arrows are needed. A variable affecting only the outcome, such as a genuine risk factor nobody consulted when prescribing, costs you precision but does not bias the comparison. A variable affecting only the assignment, such as a coin, is equally harmless. The damage requires a common cause.

The repair, and what it gives

Since severity was recorded, it can be adjusted for. Compute the effect within each stratum, then average those over one common case mix instead of the mix each arm happened to have:

$$0.5 \times 0.400 + 0.5 \times 0.700 = 0.550 \qquad\text{against}\qquad 0.5 \times 0.350 + 0.5 \times 0.650 = 0.500$$

A difference of exactly $+0.05$ - the same five points visible in each stratum, and the opposite sign to the naive $-0.05$.

Why this is not a sample-size problem

This is the part worth internalising, because it separates a statistical difficulty from a causal one.

Sampling noise shrinks as data accumulates. Confounding does not. It is a property of how the data came to exist, so collecting a million more patients under the same prescribing habits produces a tighter and tighter confidence interval around the wrong answer - and around the wrong sign.

Nothing computed from the table announces which of the two estimates is causal. The naive and adjusted numbers are both correct arithmetic on the same data. What distinguishes them is knowledge of how treatment was assigned, and that comes from outside the data.

What randomisation buys

Now take a similar world, not the one in the table: half the patients severe, untreated recovery $0.5$ for the severe and $0.8$ for the mild, and a true benefit fixed at exactly $+0.1000$ by construction. Work it out twice, changing only who decides on treatment - doctors who treat 80% of the severe and 20% of the mild, or a coin:

Those are the exact population values; a 1,500-trial simulation of the same world lands a few thousandths away, near $-0.082$ and $+0.098$.

The treatment is identical in both. When doctors assign, severity flows into the treatment variable and the estimate reverses. When a coin assigns, no patient characteristic can influence the arm, and the naive comparison recovers the effect.

That is the whole argument for randomisation, and it is stronger than adjustment in a specific way: adjustment can only handle confounders you know about and have recorded, and the claim that your list is complete cannot be checked from inside the data. Randomisation breaks the arrow from every unmeasured cause at once, without needing to name any of them.

It is worth being precise about the promise. Randomisation balances the arms in expectation; one small trial can still be unlucky, which is why trials are sized deliberately and baseline tables are inspected. And it protects exactly one point in the chain: assignment. If patients drop out afterwards for reasons connected to their treatment, the groups that remain are confounded again, which is why analyses are specified as intention-to-treat.

The world is specified exactly enough that neither row needs simulating, and the figure below computes both: -0.08 when severity decides and +0.10 when a coin does, which are the exact values in the table above. The slider is the exercise. Bring the two assignment probabilities together and the first row becomes the second, because the bias is exactly the severity imbalance times the 0.3 gap in baseline recovery. Note what that never required: a fair coin. Any common probability does it.

The adjustment that makes things worse

The lesson so far sounds like "adjust for what you can". That is where many analyses go wrong, because some adjustments create bias.

Take two causes that are genuinely independent - each occurring with probability $1/2$, with no relationship whatsoever. Suppose a case is admitted to a study when either cause is present.

Among the admitted:

• the probability of the first cause is $2/3$
• but given that the second cause is also present, it drops to $1/2$

The joint probability among the admitted is $1/3$, while the product of the marginals is $4/9$. Independence is gone.

Nothing changed in the world. Conditioning on a common effect - a collider

• manufactured a dependence between causes that had none. Intuitively: once you know a case was admitted, learning that one cause was present explains the admission and makes the other less likely.

This is why "control for everything you measured" is not sound advice, and why adding variables to a regression is a causal claim rather than a safety measure.

The figure below checks a choice of controls from both directions at once. The graph side runs d-separation over the enumerated paths; the arithmetic side runs the adjustment formula over the joint distribution. Neither consults the other, and they never disagree. Add the collider and watch a criterion fail and an estimate move on the same click - then add the confounder as well, and watch the damage stay, because a wrong control is not cancelled by a right one.

The rule, and its price

Drawing the assumptions as a graph makes the distinction readable. Three shapes cover most cases:

• fork ($T \leftarrow Z \rightarrow Y$): a common cause. Adjust for $Z$ - this is confounding.
• chain ($T \rightarrow M \rightarrow Y$): a mediator. Adjusting for $M$ removes part of the very effect you are measuring.
• collider ($T \rightarrow C \leftarrow Y$): a common effect. Adjusting for $C$ creates a spurious association.

The back-door criterion turns this into a rule: adjust for a set of variables that blocks every path into the treatment from behind and contains no descendant of the treatment, so that no mediator is adjusted away and no collider opened.

When the assumptions hold, the result is exact rather than approximate. In a world built so the true effect is exactly $0.200000$, the naive comparison gives $0.375325$ - overstating by $0.175325$ - and the back-door adjustment returns $0.200000$ exactly. Both figures are computed in exact fractions with no sampling, so that gap is bias, not noise.

The price is honesty about what the arrows rest on. Correlation is symmetric and causation is not, so several different graphs can generate the same table of numbers. The graph cannot be read off the data; it is brought to the data from knowledge of how things work.

That is a feature. It forces the assumption into the open where it can be argued with, instead of leaving it implicit in a list of regression controls.

What to carry away

A comparison between the treated and the untreated answers the causal question only if the groups were otherwise alike. When they were not, the estimate is not merely imprecise - it can point the wrong way, and more data will make it more confident rather than more correct.

So the most useful question about any observational result is not which test was run or how large the sample was. It is: how were these groups formed?

The training path Causal Inference works through all three arguments in detail, with every computation runnable and editable in the browser.