Contents
Cochran's Q Test: McNemar for Three or More Conditions
A practical guide to Cochran's Q test for comparing binary outcomes across three or more related conditions. Learn when to use it, how to interpret results, and post-hoc follow-ups.
Quick Hits
- •Cochran's Q generalizes the McNemar test to 3+ related conditions
- •Same subjects measured under each condition with a binary outcome
- •Q statistic follows a chi-square distribution with k-1 degrees of freedom
- •If significant, follow up with pairwise McNemar tests + Bonferroni correction
- •Common use: usability testing across multiple interface designs
Cochran's Q Test is the natural extension of the McNemar Test when you have three or more related conditions instead of just two. This guide covers the practical details.
When to Use Cochran's Q
The typical scenario: you have the same group of subjects tested under multiple conditions, and the outcome is binary (success/failure, pass/fail, yes/no).
Common applications:
- Usability testing: Can users complete a task across three interface designs?
- Content moderation: Do three different prompt templates correctly classify violations?
- Clinical trials: Do patients respond to three different treatment protocols?
- A/B/C testing with paired design: Does the same set of users succeed across three conditions?
Post-Hoc Comparisons
A significant Cochran's Q tells you that at least one condition differs, but not which ones. Follow up with:
- Pairwise McNemar tests for each pair of conditions
- Bonferroni correction: Divide your significance level by the number of comparisons. With 3 conditions, there are 3 pairs, so use = 0.05/3 = 0.017.
Example: Three Onboarding Flows
50 new users each try all three onboarding flows (order randomized). Success = completed setup within 10 minutes.
| Flow | Success Rate |
|---|---|
| Flow A (video tutorial) | 62% |
| Flow B (interactive wizard) | 78% |
| Flow C (documentation only) | 54% |
Cochran's Q = 8.4, df = 2, p = 0.015. The flows differ significantly.
Pairwise McNemar tests with Bonferroni correction:
- A vs. B: p = 0.012 (significant after correction at 0.017)
- A vs. C: p = 0.34 (not significant)
- B vs. C: p = 0.003 (significant)
Conclusion: The interactive wizard (Flow B) outperforms the documentation-only approach (Flow C), while the video tutorial (Flow A) falls in between.
See also: Comparing More Than Two Groups for an overview of multi-group comparison strategies.
References
- https://www.statisticshowto.com/cochrans-q-test/
- https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900058/
Frequently Asked Questions
What is the minimum sample size for Cochran's Q?
Can I use Cochran's Q with more than two outcome categories?
How does Cochran's Q relate to the Friedman test?
Key Takeaway
Cochran's Q is the right test when you have the same subjects measured across three or more conditions with a binary outcome. It is the multi-condition generalization of McNemar. Always follow a significant result with pairwise McNemar tests to identify which conditions differ.