StatsTest Blog
Experimental design, data analysis, and statistical tooling for modern teams. No hype, just the math.
Paired vs. Independent Data: A Diagnostic Checklist
How to determine whether your data is paired or independent, and why getting this wrong can dramatically affect your statistical power and validity.
Percentiles and Latency: Comparing P50, P95, P99 Correctly
How to properly compare percentile metrics like latency P50, P95, and P99 across groups. Learn about bootstrap inference, quantile regression, and the pitfalls of naive percentile comparisons.
Picking the Right Test to Compare Two Groups: A Decision Framework
A comprehensive guide to choosing between t-tests, Mann-Whitney, bootstrap, and other methods when comparing two groups. Covers continuous, binary, and count data with practical decision trees.
Poisson vs. Negative Binomial: Modeling Counts and Rates
A practical guide to choosing between Poisson and negative binomial regression for count data. Learn to detect overdispersion, handle excess zeros, and interpret rate ratios correctly.
Post-Hoc Tests: Tukey, Dunnett, and Games-Howell Decision Tree
How to choose the right post-hoc test after ANOVA. Covers Tukey's HSD, Dunnett's test, Games-Howell, Scheffé, and provides a clear decision tree for selection.
Power Analysis Without Cargo Culting: Traps and Practical Heuristics
A practical guide to statistical power analysis that avoids common pitfalls. Learn when standard power calculations mislead, how to think about sample size decisions, and practical heuristics for real-world experimentation.
Practical Significance Thresholds: Defining Business Impact Before You Analyze
Learn how to set meaningful thresholds for practical significance before running experiments. Covers MDE, business context, ROI-based thresholds, and the difference between statistical and practical significance.
Pre-Analysis Checklist: Green, Yellow, and Red Flags for Analysts
A practical pre-flight checklist before running statistical analyses. Covers data quality, assumption checks, and common pitfalls that can derail your analysis.
Pre-Registration Lite for Product Experiments: A Pragmatic Workflow
A lightweight pre-registration process that works in fast-moving product teams. Document your analysis plan in 15 minutes and build credibility through transparency.
Ratio Metrics (CTR, Conversion): Why They're Tricky and Stable Alternatives
Why ratio metrics like CTR and conversion rates require special statistical treatment. Learn about variance estimation, the delta method, and when to use alternative approaches.
Regression for Analysts: From Comparison to Causal Insight
A comprehensive guide to regression analysis for product and data analysts. Learn when to use linear, logistic, and count regression, how to diagnose problems, interpret coefficients correctly, and avoid common pitfalls that lead to misleading conclusions.