Cochran's Q Test

The StatsTest Flow: Difference >> Proportional/Categorical >> Paired/Repeated Measurements >> Three or more conditions

Not sure this is the right statistical method? Use the Choose Your StatsTest workflow to select the right method.

What is Cochran's Q Test?

Cochran's Q Test is a non-parametric statistical test used to determine whether the proportions of a binary outcome differ across three or more related groups. It is the extension of the McNemar Test to more than two conditions.

The test is used when the same subjects are measured under multiple conditions (repeated measures) and the outcome is binary. For example, testing whether the same users can complete a task across three different UI designs.

Cochran's Q Test is also called Cochran's Test for Related Proportions or the Repeated Measures Chi-Square Test (for binary data).

Assumptions for Cochran's Q Test

The assumptions for Cochran's Q Test include:

1. Binary Outcome
2. Related Samples (Repeated Measures)
3. Sufficient Sample Size
4. Random Sampling

Binary Outcome

Each observation must be coded as success (1) or failure (0). The test compares the proportion of successes across conditions.

If your outcome is continuous with repeated measures, use a One-Way Repeated Measures ANOVA (normal data) or Friedman Test (non-normal data) instead.

Related Samples

The same subjects must be measured under each condition. Each row in the data represents one subject, and each column represents one condition.

If groups are independent (different subjects in each group), use the Chi-Square Test of Independence instead.

Sufficient Sample Size

The chi-square approximation used by Cochran's Q works well when the total number of successes is not too small. A common guideline is at least 10 subjects, though more is better.

Random Sampling

Subjects should be randomly sampled from the population of interest for the results to be generalizable.

When to use Cochran's Q Test?

1. Your outcome is binary (success/failure)
2. You have three or more related conditions
3. The same subjects are measured under each condition
4. You want to know if the success rate differs across conditions

If you have only two related conditions, use the McNemar Test instead.

Cochran's Q Test Example

Subjects: 50 customer service representatives. Conditions: Three different knowledge base interfaces (A, B, C). Outcome: Whether the rep successfully resolved a test case within 5 minutes (yes/no).

Each rep attempts the same test case on each of the three interfaces (order counterbalanced). We record whether they succeeded on each one. The success rates are: Interface A = 64%, Interface B = 78%, Interface C = 72%.

Cochran's Q test produces Q = 7.2, df = 2, p = 0.027. Since p < 0.05, we conclude that the interfaces differ in success rate. Follow-up pairwise McNemar tests (with Bonferroni correction) reveal that Interface B is significantly better than Interface A (p = 0.008), while the other pairwise comparisons are not significant.