StatsTest Blog
Experimental design, data analysis, and statistical tooling for modern teams. No hype, just the math.
A/B Testing Statistical Methods for Product Teams: The Complete Guide
A comprehensive guide to statistical methods for A/B testing in product development. Learn when to use z-tests, chi-square, sequential testing, CUPED, and how to handle the real-world messiness of experimentation.
Choosing the Right Test for Conversion Rates: Z-Test, Chi-Square, or Fisher's Exact
A practical decision framework for selecting between z-test, chi-square test, and Fisher's exact test when comparing conversion rates in A/B experiments.
Clustered Experiments: Geo Tests, Classrooms, and Independence Violations
How to handle A/B tests where observations aren't independent—geo experiments, marketplace tests, social features, and other settings where users are clustered or connected.
CUPED and Variance Reduction: When It Helps and When It Backfires
Learn how CUPED (Controlled-experiment Using Pre-Experiment Data) can dramatically reduce variance in A/B tests, when to use it, and the pitfalls that can make it backfire.
Minimum Detectable Effect and Sample Size: A Practical Guide
Learn how to calculate the minimum detectable effect for your A/B test, determine required sample sizes, and understand the tradeoffs between statistical power and practical constraints.
Multiple Experiments: FDR vs. Bonferroni for Product Teams
How to manage false discoveries when running many A/B tests simultaneously. Learn when to use Bonferroni, Benjamini-Hochberg FDR, and when corrections aren't needed.
Non-Normal Metrics: Bootstrap, Mann-Whitney, and Log Transforms
How to analyze A/B test metrics that aren't normally distributed—heavy-tailed revenue, skewed engagement, and other messy real-world data. Covers bootstrap methods, Mann-Whitney U, and when transformations help.
Sample Ratio Mismatch: Detection, Root Causes, and Solutions
How to detect sample ratio mismatch (SRM) in A/B tests, understand its common causes, and what to do when your experiment groups have unexpected sizes.
Sequential Testing: How to Peek at P-Values Without Inflating False Positives
Learn how sequential testing methods let you monitor A/B test results as data accumulates while maintaining valid statistical guarantees. Covers group sequential designs, always-valid inference, and practical implementation.