Skip to content
Kudos AI
enfrar

Peeking and the Dashboard of Metrics

A test with no effect at all, watched ten times and stopped when it looks good, comes out significant 19% of the time; the same arithmetic applied to twenty independent metrics gives 64%. Both, and what to do instead.

IntermediateModule 230 min · 120 XP
A test statistic wandering as data accumulates and crossing a fixed threshold on one of ten looks, then a dashboard of twenty null metrics lighting up one at a time until something is significant.

This is a premium lesson

Sign in and enrol to read the full lesson, run the code, take the quiz, and earn XP toward the path badge.

Sign inCreate account
← Back to Experimentation and A/B Testing