Attribution tells you which channel gets credit for a conversion. But does the channel actually drive the conversion, or would it have happened anyway? Incrementality testing answers this question by running experiments: expose some users to the channel (treatment group), don't expose others (control group), measure the difference in conversion rates. The difference is the true incremental impact. This chapter covers incrementality testing and when to use it.
Attribution vs. Incrementality
Attribution: of people who converted, which channel touched them last? Simple to measure, but doesn't prove causation.
Incrementality: if we remove this channel entirely, how much revenue would we lose? This proves causation and measures true impact.
Example: 100 people converted. Google Ads last-click = 60 get credit. But incrementality test shows: if we didn't run Google Ads, 50 of those 60 would have still converted (through other channels). True incrementality of Google Ads = 10 conversions, not 60.
Types of Incrementality Tests
Geo-based testing
Run ads in some geographies (treatment), not others (control). Compare conversion rates. Example: run Google Ads in San Francisco (treatment), not in Berkeley (control). If SF has higher conversion rate, ads are incremental.
Advantage: simple, fast. Disadvantage: geography differences may confound results (SF is wealthier, skews conversion rate higher).
Holdout groups
Run ads to everyone except a random 10% (control). Compare conversion rates: ad-exposed users vs. holdout users. Difference is incrementality.
Advantage: accurate, controls for confounds. Disadvantage: requires large budget (we're not targeting 10% of our audience to measure impact).
Time-based testing
Run ads for some time periods (treatment), not others (control). Example: run paid search Mon-Thurs, not Fri-Sun. Compare conversion rates. If Mon-Thurs is higher, ads are incremental.
Advantage: simple. Disadvantage: day-of-week confounds (weekends are naturally lower conversion).
Running an Incrementality Test
Step 1: Define hypothesis. "Google Ads drives incremental conversions beyond baseline."
Step 2: Choose test type. Holdout groups is most accurate; use that if budget allows.
Step 3: Size your test. Need enough sample to detect difference. If baseline conversion = 5%, and you want to detect 2% lift, you need ~10,000 per group. Consult statistician or use power calculator.
Step 4: Run test for 2-4 weeks. Longer tests reduce variance from daily fluctuations.
Step 5: Analyze results. Is difference statistically significant? (p < 0.05). If yes, you've proven incrementality.
Challenges in Incrementality Testing
Cost: holdout groups mean losing revenue on purpose (we don't target 10% of our audience). Only worth it for high-spend channels where the insight is valuable.
Cannibalization: if user doesn't see your ad, they may use organic search or direct traffic instead. Incrementality captures this (it's included in control group behavior).
Long-term effects: ads may have brand effect that shows up days or weeks later. Most tests are short-term; miss long-term impacts.
How do I decide whether to run an incrementality test?
Incrementality is expensive (holdout costs money). Run tests for: high-spend channels (Google Ads, Facebook Ads), where attribution is ambiguous, where you're making large budget decisions. Don't test: small channels, where attribution is clear. Budget: $10k-50k+ per test. Reserve for highest-impact questions.
What sample size do I need for an incrementality test?
Depends on baseline conversion rate and lift you want to detect. Rule of thumb: need 5,000-10,000 per group (control and treatment). If baseline is 5% and you want to detect 2% lift (7% new rate), need ~10,000 per group. Use power calculator (search 'statistical power calculator') to get exact number for your scenario.
How long should I run an incrementality test?
2-4 weeks minimum (to average out daily variation). 4-8 weeks better (captures weekly patterns). Longer tests = higher cost but more reliable. If your sales cycle is 30 days, run test for 30+ days to capture full cycle.
Can I use incrementality testing to measure channel interactions (how one channel helps another)?
Yes, but more complex. Test design: (1) control (no channels), (2) treatment A (channel A only), (3) treatment B (channel B only), (4) treatment A+B (both channels). Compare results. If A+B > A + B, there's synergy. This requires larger sample sizes and more complex analysis.
What's the difference between a holdout group and a control group in testing?
Holdout: part of your customer base that doesn't see the channel (on purpose). Control group: in an experiment, the group that doesn't receive treatment (for comparison). Same concept, different terminology. Holdout is continuous (ongoing holdout group); control is for specific test duration.
How do I present incrementality test results to leadership?
Show: baseline conversion (5%), treatment conversion (6%), lift (1 percentage point, 20% relative increase). Show confidence interval: lift is 0.5-1.5 percentage points (we're confident about this range). Budget implication: if incrementality is 20% lower than attribution claimed, recalculate ROI. This justifies the test cost with better decision-making.