Which Wrong Distribution Do You Want?

The target $P$ is an even mixture of two unit Gaussians at $-4$ and $+4$. It has mean $0$, standard deviation $4.1231$, and essentially no probability anywhere near its own mean.

The model $Q$ is a single Gaussian. It cannot represent $P$, which is the normal situation: the question is what "as close as possible" means.

The Kullback-Leibler divergence gives two answers, because it is not symmetric.

A. Two fits

$$D(P \parallel Q) = \int p \log \frac{p}{q}, \qquad D(Q \parallel P) = \int q \log \frac{q}{p}.$$

Minimise each over the mean and standard deviation of $Q$:

The first fit spans both modes. Its standard deviation, 4.1231, is exactly the standard deviation of $P$ itself, $\sqrt{1 + 16} = \sqrt{17}$: minimising $D(P \parallel Q)$ over Gaussians matches the mean and variance of the target, and here that puts the centre at the point where $P$ has almost no mass at all. The second fit lands on one mode, within a thousandth: mean 3.9996 against a true mode at 4, standard deviation 1.0008 against a true 1. The small offsets are the other component's tail, reaching faintly across the gap.

Both answers are correct. They are answers to different questions.

B. Why 0.6931

The reverse-KL value is $\ln 2 = 0.6931$ to four decimals, and that is not a coincidence. When $Q$ sits on one component, $P \approx \tfrac{1}{2}Q$ throughout the region where $Q$ has mass, since the other component contributes almost nothing there. So

$$D(Q \parallel P) \approx \int q \log \frac{q}{\tfrac{1}{2} q} = \log 2.$$

The "almost" can be made exact. With $Q = N(4, 1)$, the target is $p = \tfrac{1}{2} q \,(1 + e^{-8x})$, so

$$D(Q \parallel P) = \ln 2 - \mathbb{E}_q\!\left[\ln\left(1 + e^{-8x}\right)\right],$$

and the correction, the other component's tail reaching across the gap, is about $0.0001$. The fitted value is $0.693053$ against $\ln 2 = 0.693147$.

The penalty for ignoring half the target is, up to that sliver, the one bit of information needed to say which half was kept. That is a clean way to see what reverse KL does and does not charge for: it charges for putting mass where the target has none, and charges nothing at all for failing to cover the target's other regions.

C. Each fit is a disaster by the other measure

The reverse fit scores 15.2799 under forward KL, twenty-one times worse than the forward fit's own 0.7236. This is not a small preference between two nearly equivalent criteria; the two objectives disagree about which model is usable.

The asymmetry has a direction that is easy to remember:

• Forward, $D(P \parallel Q)$, is mean-seeking. The integral is weighted by $p$, so wherever $P$ has mass and $Q$ does not, the ratio $p/q$ blows up. $Q$ is forced to cover everything $P$ does, at the cost of covering a great deal that $P$ does not.
• Reverse, $D(Q \parallel P)$, is mode-seeking. The integral is weighted by $q$, so regions where $P$ has mass and $Q$ does not cost nothing. $Q$ is free to ignore most of $P$, provided it avoids putting mass where $P$ has none.

D. Which one you are already using

Both are in daily use, usually without the direction being stated.

• Maximum likelihood is forward KL. Minimising cross-entropy against the data distribution minimises $D(P \parallel Q)$ up to a constant. That is why a maximum-likelihood fit spreads: it is being penalised for every region of the data it fails to cover.
• Variational inference is reverse KL. The evidence lower bound minimises $D(Q \parallel P)$, which is why a variational posterior is famously overconfident and latches onto one mode.
• Distillation, RLHF and their relatives choose a direction too, and the choice shows up as whether the student hedges across the teacher's options or commits to one of them.

The practical rule: if leaving out part of the target is the expensive mistake, use forward. If claiming things the target rules out is the expensive mistake, use reverse. And when someone reports "the KL divergence" as one number, the first question is which way round it was taken.