Tagged “generalization”

2 articles.

14 September 20264 min readStatistical Learning Theory

The Theorem That Says Nothing About Your Problem

Averaged over all 256 functions from three bits to one, a nearest-neighbour learner and a learner built to be wrong on purpose both score exactly 0.500000 off the training set. That is the no free lunch theorem, it is exactly true, and the moment the average is restricted to the six functions that depend on a single bit the two separate to 0.333333 and 0.666667.

Machine LearningMathematics

14 September 20264 min readStatistical Learning Theory

One Parameter, Infinite Capacity

A classifier with exactly one real parameter fits all 1,048,576 labellings of twenty points, every time, and predicts a twenty-first at 0.5038 accuracy over twenty thousand trials. Counting parameters measures neither an upper nor a lower bound on what a model class can fit, which is why capacity has to be measured some other way.

Machine LearningMathematics