Mediation Analysis: Does Feature X Work Through Mechanism Y?

TL;DR

When a product change moves a metric, the natural next question is: why? Mediation analysis answers this by decomposing the total treatment effect into a direct effect and an indirect effect that flows through a hypothesized mechanism. It is the formal framework for answering "does Feature X work because it increases Engagement, or for some other reason?" This post covers the modern causal approach to mediation, the assumptions that make it valid, and how product teams can use it to make better decisions about what to build next.

Why Mechanisms Matter in Product Analytics

Knowing that a feature works is valuable. Knowing how it works is more valuable. Consider a scenario where you launched an in-app tutorial and observed a 15% lift in user activation. Two possible mechanisms:

1. The tutorial teaches users how to use the product, reducing confusion.
2. The tutorial forces users to interact with key features, creating habit loops.

If mechanism 1 dominates, you should invest in better educational content. If mechanism 2 dominates, you should invest in guided workflows and activation checklists. The treatment is the same; the optimal next step depends on the mechanism.

Mediation analysis formalizes this decomposition.

The Causal Framework

Setup

• Treatment (T): The intervention (e.g., tutorial shown vs. not shown).
• Mediator (M): The hypothesized mechanism (e.g., feature exploration in the first session).
• Outcome (Y): The metric you care about (e.g., 30-day activation).

The causal DAG looks like:

T --> M --> Y
T ---------> Y

The total effect of $T$ on $Y$ decomposes into:

• Indirect effect (IE): The portion of the effect that flows through $M$. ($T \to M \to Y$)
• Direct effect (DE): Everything else. ($T \to Y$ not through $M$)

$\text{Total Effect} = \text{Direct Effect} + \text{Indirect Effect}$

Modern Causal Definitions

Using potential outcomes notation (Imai, Keele, Tingley, 2010):

• Average Causal Mediation Effect (ACME): $E[Y(1, M(1)) - Y(1, M(0))]$ -- the effect of changing the mediator from what it would be under control to what it would be under treatment, while holding treatment at treated.
• Average Direct Effect (ADE): $E[Y(1, M(0)) - Y(0, M(0))]$ -- the effect of changing treatment while holding the mediator at its control value.

These definitions avoid the pitfalls of the classic Baron and Kenny approach, which relies on linear models and implicitly assumes no interaction between treatment and mediator.

The Classic vs. Modern Approach

Baron and Kenny (1986) -- Outdated but Still Common

The classic approach runs three regressions:

1. Regress $Y$ on $T$: check that treatment affects the outcome.
2. Regress $M$ on $T$: check that treatment affects the mediator.
3. Regress $Y$ on $T$ and $M$: check that the mediator affects the outcome after controlling for treatment, and that the treatment coefficient shrinks.

The indirect effect is estimated as the product of coefficients: $a \times b$ (where $a$ is the effect of $T$ on $M$ and $b$ is the effect of $M$ on $Y$ controlling for $T$).

Problems: This approach assumes linear relationships, no treatment-mediator interaction, and makes strong parametric assumptions. It also conflates statistical significance of individual paths with evidence of mediation.

Modern Causal Mediation Analysis

The modern approach, based on the potential outcomes framework:

1. Defines direct and indirect effects using counterfactuals.
2. Works with any type of outcome and mediator (binary, continuous, count).
3. Allows treatment-mediator interactions.
4. Makes the identifying assumptions explicit (especially sequential ignorability).
5. Provides sensitivity analysis tools for violations of those assumptions.

Use the mediation package in R or equivalent implementations in Python.

The Critical Assumption: Sequential Ignorability

For mediation analysis to yield causal estimates, you need sequential ignorability:

1. No unmeasured confounders of T and Y. If treatment is randomized, this is satisfied by design.
2. No unmeasured confounders of M and Y, conditional on T and covariates. This is the hard part.

Even if you randomized treatment, the mediator is not randomized. Users who engage with a tutorial more deeply may differ from those who skim it, and those differences may also affect the outcome. This is the same challenge as any observational causal inference problem: confounding of a non-randomized variable.

Sequential ignorability is strong and often implausible. This does not mean mediation analysis is useless -- it means you must:

• Be transparent about the assumption.
• Run sensitivity analysis to assess how much confounding would be needed to invalidate the indirect effect.
• Treat results as suggestive rather than definitive unless you have exceptional data.

A Product Analytics Example

Context: Your team launched a "social proof" feature that shows new users how many others have completed onboarding. An A/B test showed a 10% lift in 14-day retention. You hypothesize the mechanism is increased onboarding completion.

Variables:

• $T$: Social proof feature (shown vs. not shown), randomized.
• $M$: Onboarding completion rate within 3 days.
• $Y$: 14-day retention.
• Covariates: Sign-up source, device type, country.

Analysis:

1. Estimate the mediator model: logistic regression of onboarding completion on treatment and covariates.
2. Estimate the outcome model: logistic regression of 14-day retention on treatment, onboarding completion, and covariates.
3. Use the mediation package to compute ACME and ADE with quasi-Bayesian confidence intervals.

Results (hypothetical):

• Total effect: +10% retention (from the A/B test).
• ACME (indirect via onboarding): +7% (95% CI: +4% to +10%).
• ADE (direct): +3% (95% CI: -1% to +6%).

Interpretation: About 70% of the social proof feature's effect on retention flows through increased onboarding completion. The remaining 30% operates through other channels (perhaps a general sense of social belonging).

Decision: Invest in further improving onboarding completion, since that is the primary mechanism.

Sensitivity Analysis

Because sequential ignorability is untestable, always run a sensitivity analysis. The Imai, Keele, and Yamamoto (2010) approach parameterizes the sensitivity in terms of $\rho$ -- the correlation between the residuals of the mediator and outcome models.

At $\rho = 0$ (no unmeasured confounding), you get your baseline ACME. As $\rho$ increases (stronger unmeasured confounding), the ACME changes. The question is: at what value of $\rho$ does the ACME cross zero?

If it takes $\rho > 0.5$ to nullify the indirect effect, you have some confidence. If $\rho = 0.1$ is enough, the result is fragile.

Common Pitfalls

Confusing mediation with moderation. Mediation is about how; moderation is about for whom or when. Running subgroup analyses (moderators) does not tell you about mechanisms (mediators).

Mediator measured post-treatment but pre-outcome ambiguity. Ensure the mediator is temporally between treatment and outcome. If the timeline is unclear, the causal ordering is unclear.

Conditioning on a collider. If a variable is caused by both the mediator and the outcome, including it in the model introduces confounding where none existed. Check your DAG before adding controls.

Ignoring treatment-mediator interaction. The classic approach assumes the effect of $M$ on $Y$ is the same regardless of treatment status. The modern approach allows for this interaction, and ignoring it can bias both direct and indirect effect estimates.

Over-interpreting proportions. Saying "70% of the effect is mediated" sounds precise, but the proportion mediated is a ratio of estimates, each with uncertainty. The confidence interval on the proportion can be very wide. Report point estimates with intervals.

When to Use Mediation Analysis

Use mediation analysis when:

• You have a well-defined mechanism hypothesis before looking at data.
• The treatment is randomized or you can make a credible case for no confounding of $T$.
• You have a measurable mediator that is temporally between treatment and outcome.
• You are willing to run sensitivity analysis and report it transparently.

Avoid it when:

• You are fishing for mechanisms without a prior hypothesis.
• The mediator is measured simultaneously with the outcome.
• You cannot articulate what unmeasured confounders of $M$ and $Y$ might exist.

For the broader causal inference toolkit, see our overview of methods for when experiments aren't possible.