Edit two distributions side by side and read off entropy, cross entropy, and KL divergence in bits, with the identity H(p, q) = H(p) + KL(p ‖ q) held on screen.
Everything in bits (base 2).
H(p, q) = H(p) + KL(p ‖ q) → 1.7370 = 1.0000 + 0.7370
Maximum for 2 outcomes: 1.0000 bits
Entropy is highest when every outcome is equally likely and falls to zero once one outcome is certain. Cross entropy is what a model is trained to minimise; it splits exactly into the entropy of the data - which no model can remove - plus the KL divergence, the part caused by believing the wrong distribution.
Runs entirely in your browser. Nothing you enter is uploaded or stored.
Measuring uncertainty in bits: Shannon entropy and why the logarithm is base 2, information gain worked on a split, and how cross-entropy and KL divergence relate to entropy and to the loss functions used to train classifiers.
How recursive binary splitting builds a tree, why the Gini index beats accuracy as a splitting criterion, and how bagging and random forests turn a high-variance learner into a strong one, with the split arithmetic worked out.