Sample Ratio Mismatch (SRM) Diagnostic Analyzer
Diagnose Sample Ratio Mismatch (SRM)—a fundamental experimental integrity flaw where observed traffic splits differ significantly from intended allocation ratios, invalidating test conclusions.
Actual count of visitors or users assigned to control
Actual count of visitors or users assigned to treatment
Relative expected weight for Control (e.g. 1 for 50/50 split)
Relative expected weight for Variant (e.g. 1 for 50/50 split)
Validation status based on standard p < 0.001 threshold
How to Use This Calculator
- Enter the actual user count in the Control group.
- Enter the actual user count in the Variant group.
- Confirm target allocation ratio (1:1 is default 50/50).
- Review the SRM status and chi-square p-value.
Formula & Calculation Methodology
Performs Pearson’s Chi-Square goodness-of-fit test against theoretical allocations. A p-value below 0.001 signals overwhelming evidence of a systemic assignment malfunction.
Worked Example
Evaluating 50,450 Control vs 49,550 Variant (100k total, 50/50 target)
Severe mismatch: 52,000 Control vs 48,000 Variant (100k total)
Frequently Asked Questions
Why does SRM invalidate an entire A/B test?↓
If users are systematically missing from one branch, the two groups are no longer comparable (e.g. ad blockers or safari users might be crashing in variant B). Concluding anything about conversion differences is deeply compromised.
Why use p < 0.001 instead of p < 0.05 for SRM?↓
With huge online sample sizes (hundreds of thousands of users), tiny inconsequential routing jitter can cross p = 0.05. Industry consensus (Microsoft, Google) uses p < 0.001 to flag genuine engineering defects.
What causes Sample Ratio Mismatch in production?↓
Common causes include redirect latency discrepancies, bot filters acting unevenly on one variant, tracking tag placement bugs, or client-side JavaScript crashes before the logging beacon fires.
Can SRM be fixed by re-weighting or normalizing data?↓
No. You cannot normalize away missing users because you do not know the unobserved characteristics of the dropped cohort. The test must be fixed and re-run.
Related Calculators
False Positive Risk Calculator for A/B Test Peeking
Open calculator →Sequential Testing Duration and Boundary Calculator
Open calculator →Minimum Reliably Detectable Effect (MRDE) Calculator
Open calculator →Dunnett's Correction A/B/n Sample Size Estimator
Open calculator →Sample Size Calculator
Open calculator →Source Citation & Disclaimer
Informational tool only; not licensed tax, legal, or financial advice.