Tagged “entropy”

2 articles.

11 September 20267 min readInformation Theory

The Bound That Is Actually Reached

Entropy is not a summary of a distribution but a floor that the best code meets to the last decimal, the surcharge for using the wrong distribution is exactly the loss every classifier already minimises, and mutual information puts a hard ceiling on everything downstream of a sensor. Three results, each unusually sharp.

MathematicsMachine Learning

29 August 202610 min readProbability Foundations

Entropy and Information

Measuring uncertainty in bits: Shannon entropy and why the logarithm is base 2, information gain worked on a split, and how cross-entropy and KL divergence relate to entropy and to the loss functions used to train classifiers.

Information TheoryProbabilityMathematics