False Positive Risk Calculator for A/B Test Peeking
Quantify the severe statistical penalty and false positive inflation caused by "peeking" (optional stopping)—checking an ongoing A/B test repeatedly and stopping as soon as p < 0.05 appears.
The intended nominal Type I error threshold
How many times the experiment dashboard is checked during the run
Actual probability of falsely detecting an effect where none exists
How to Use This Calculator
- Select your planned nominal significance level (0.05 default).
- Enter the number of times you plan to inspect test significance.
- View the inflated false positive rate and loss of statistical confidence.
Formula & Calculation Methodology
Repeatedly evaluating random walk noise guarantees that p will cross 0.05 at some point even when the null hypothesis is completely true. The false positive rate scales logarithmically with the number of peeks K.
Worked Example
Checking an A/B test 10 times at nominal α = 0.05
Checking an A/B test every day for 30 days (K = 30)
Frequently Asked Questions
Why does peeking increase false positives?↓
A/B test metrics fluctuate due to random sampling variance. If you test 10 times, you give random chance 10 opportunities to cross the significance line, effectively multiplying your risk.
How can I monitor tests safely without peeking penalties?↓
Use formal Sequential Testing (like O’Brien-Fleming or Always Valid p-values via SPRT), or commit to a fixed sample size and look only once.
Is it safe to stop a test early if the result is overwhelmingly negative?↓
Stopping early for catastrophic harm is valid (futility stopping), provided safety guardrails are established prior to launch.
What is a limitation of this approximation?↓
The formula models equally spaced looks on independent normally distributed increments; continuous real-time streaming dashboards exhibit even higher inflation.
Related Calculators
Sequential Testing Duration and Boundary Calculator
Open calculator →Minimum Reliably Detectable Effect (MRDE) Calculator
Open calculator →Dunnett's Correction A/B/n Sample Size Estimator
Open calculator →Sample Ratio Mismatch (SRM) Diagnostic Analyzer
Open calculator →Sample Size Calculator
Open calculator →Confidence Interval Calculator
Open calculator →Source Citation & Disclaimer
Informational tool only; not licensed tax, legal, or financial advice.