StatsTest Blog
Experimental design, data analysis, and statistical tooling for modern teams. No hype, just the math.
Pre-Registration Lite for Product Experiments: A Pragmatic Workflow
A lightweight pre-registration process that works in fast-moving product teams. Document your analysis plan in 15 minutes and build credibility through transparency.
Ratio Metrics (CTR, Conversion): Why They're Tricky and Stable Alternatives
Why ratio metrics like CTR and conversion rates require special statistical treatment. Learn about variance estimation, the delta method, and when to use alternative approaches.
Regression vs. t-Test vs. ANOVA: The Unifying View (and When the Simpler Tool Suffices)
Understand how t-tests, ANOVA, and regression are all the same underlying model. Learn when to use the simpler approach and when regression's flexibility is worth it.
Reporting Templates: Stakeholder Language Without Overclaiming
Ready-to-use templates for presenting statistical results to non-technical stakeholders. Learn to communicate effect sizes, uncertainty, and practical significance without oversimplifying or overclaiming.
Robust Standard Errors: When to Use Them (and When to Use by Default)
A practical guide to heteroscedasticity-robust and cluster-robust standard errors. Learn when standard errors are wrong, which corrections to apply, and whether to use robust standard errors by default.
Robust Statistics Toolbox: Trimmed Means, Winsorization, and Rank Methods
A practical guide to robust statistical methods that work without normality assumptions. Learn when to use trimmed means, Winsorization, M-estimators, and rank-based tests.
Sample Ratio Mismatch: Detection, Root Causes, and Solutions
How to detect sample ratio mismatch (SRM) in A/B tests, understand its common causes, and what to do when your experiment groups have unexpected sizes.
Sequential Testing: How to Peek at P-Values Without Inflating False Positives
Learn how sequential testing methods let you monitor A/B test results as data accumulates while maintaining valid statistical guarantees. Covers group sequential designs, always-valid inference, and practical implementation.
Statistically Significant but Meaningless: Practical Thresholds for Evals
A 0.5% accuracy improvement with p<0.001 is real but worthless. Learn how to distinguish statistically significant from practically meaningful in model evaluation.
Time-to-Event Sample Size: Practical Approximations
A practical guide to sample size calculations for survival studies. Learn how to power time-to-event analyses, what drives the sample size, and practical approximations for retention experiments.
Data Transformations: When Log, Sqrt, and Box-Cox Help vs. Mislead
A practical guide to data transformations in statistical analysis. Learn when transformations fix problems, when they create new ones, and how to interpret results correctly.
Testing Trends Across Ordered Groups: Jonckheere-Terpstra and Alternatives
When your groups have a natural order (dose levels, experience tiers, usage intensity), standard ANOVA ignores this structure. Learn about trend tests that leverage ordering for more power.