The Baseline Problem: Why You Need a Control Group
Understanding what would happen if you did nothing—the foundation of measuring impact.
You run a campaign. 100 people sign up.
You think: “Great! The campaign generated 100 signups.”
But what if those 100 people would have signed up anyway?
What if, without the campaign, your baseline signup rate would have been 80?
Then the campaign actually generated 20 incremental signups, not 100.
This is the baseline problem.
What Is a Baseline?
A baseline is what would happen if you did nothing.
If you didn’t run the campaign, how many people would sign up? That’s your baseline.
The true impact of the campaign is:
Incremental impact = Actual - Baseline
If 100 sign up with campaign and 80 would sign up without, the campaign’s real impact is 20.
Why Baselines Matter
Most businesses don’t measure baselines. They measure totals.
Campaign generated 100 signups. Email generated 200 signups. Ad platform generated 150 signups.
But you don’t know how many people would have signed up anyway in the absence of all marketing.
Maybe your product is so good that people find you naturally. Maybe 50 people per week sign up just from word of mouth and organic search.
If that’s the case:
- Campaign (100 signups) is actually only 50 incremental (100 - 50 baseline)
- Email (200 signups) is actually only 150 incremental (200 - 50 baseline)
- Ad platform (150 signups) is actually only 100 incremental (150 - 50 baseline)
But if you’re looking at totals, you think email is the winner (200 > 100 > 150).
When you account for baseline, the ranking changes.
How to Measure Baseline
Method 1: Historical average (if nothing changed)
In a week with no marketing activity, how many signups do you get?
If you get 50 signups in a quiet week, that’s your baseline. Any signups above 50 are from your marketing.
This assumes:
- Nothing else changed (product, competition, seasonality)
- The quiet week is representative
Method 2: Holdout group (control group)
Run your campaign on 90% of your audience. Hold out 10% and don’t expose them to the campaign.
Campaign group: 100 signups Holdout (control) group: 12 signups (proportional to 10% of audience)
If you expected 12 signups from the holdout group (proportional to campaign group if nothing happened), then all 100 are incremental from the campaign.
But if signups should be higher: Maybe you expected 25 from the holdout group (if baseline is high).
Then incremental impact is 100 - 25 = 75 signups from the campaign.
The holdout group tells you your baseline.
Method 3: Matched cohorts
You have customers who were exposed to your campaign and customers who weren’t (maybe they’re in a different geography where the campaign didn’t run).
Compare them:
- Campaign group: 2% conversion rate
- Control group (no campaign): 1.5% conversion rate
- Incremental impact: 0.5 percentage points
The control group’s 1.5% is your baseline. It represents what would have happened without the campaign.
The Seasonality Baseline
Sometimes baseline changes seasonally.
Summer (June-August): Natural baseline might be 50 signups/week (summer break, slower business)
Winter (November-December): Natural baseline might be 150 signups/week (holiday season, people shopping)
If you run a campaign in July and get 100 signups, the incremental impact is only 50 (100 - 50 baseline).
If you run the same campaign in December and get 150 signups, the incremental impact is only 0 (150 - 150 baseline, or it’s negative if the campaign was resource-intensive).
Without accounting for baseline seasonality, you’d think the December campaign was worse (lower absolute signups) when it was actually non-incremental.
The Trend Baseline
Business grows. Your baseline might be increasing.
January baseline: 50 signups February baseline: 55 signups (5% growth) March baseline: 60 signups (5% growth) April baseline: 63 signups (5% growth)
If you run a campaign in April and get 100 signups, the incremental impact is:
100 - 63 (baseline) = 37 signups
But if you ignore trend and assume baseline is 50 (January), you’d calculate:
100 - 50 = 50 signups
You’d overestimate the campaign’s impact by 35%.
How Marketing Platforms Lie About Baseline
Your ad platform (Facebook, Google) shows you how many people converted “within X days of seeing your ad.”
They claim credit for all of them.
But they’re not measuring baseline. They’re measuring totals.
If 100 people see your ad and 2 convert, Facebook reports: “2 conversions from your ad.”
What Facebook doesn’t tell you: Maybe 2 people would have converted anyway (without the ad).
Then the true incremental impact is 0.
Facebook’s incentive is to claim as much impact as possible. So they don’t measure baseline. They measure total conversions.
This is why we push you toward incrementality testing. It’s the only way to know the real impact.
When Baseline Doesn’t Matter
Some decisions don’t require baseline measurement.
If you’re choosing between two campaigns and you just want to know which is better, you can compare them directly without baseline.
Campaign A: 100 conversions Campaign B: 80 conversions
Campaign A is better. You don’t need baseline to make that comparison.
But if you want to know whether to scale a campaign, you do need baseline.
“Should we spend $10,000 more on Campaign A?”
That depends on: How many incremental conversions will $10,000 buy?
If Campaign A’s baseline is 90 conversions and your true incremental is only 10, then maybe it’s not worth $10,000.
But if Campaign A’s baseline is 50 and your incremental is 50, then it’s worth $10,000.
Building Baseline Measurement Into Your Process
For campaigns:
Establish a baseline for each channel in each season:
- “Facebook baseline in Q4: 1,000 conversions per week (if no ads run)”
- “Email baseline: 500 opens per send (if no campaign)”
- “Organic baseline: 50 signups per week (if no SEO work)”
Then, when a campaign runs, measure incrementally:
- Facebook campaign week: 1,500 conversions
- Incremental: 1,500 - 1,000 = 500 conversions
For features:
Before launching a feature, measure engagement baseline:
- Current session length: 5 minutes
- Current feature adoption: 20%
- Current churn rate: 5%
After launch:
- New session length: 6 minutes (1-minute increase)
- New feature adoption: 45% (25-point increase)
- New churn rate: 4% (1-point improvement)
You can now say: “Feature increased session length by 20%, adoption by 125%, and reduced churn by 20%.”
All compared to baseline.
For pricing:
Before changing prices, measure baseline conversion and revenue:
- Current price: $99
- Current conversion rate: 2%
- Current revenue: $100k
After raising to $149:
- New conversion rate: 1.5%
- New revenue: $112k
You can say: “Raising price to $149 increased revenue by 12% despite a 25% decline in conversion rate.”
That’s a real, quantifiable impact (compared to baseline).
The Takeaway
Every decision has a baseline: What would happen if you did nothing?
If you don’t measure baseline, you overestimate the impact of everything you do.
You think your campaigns are working great when they’re just riding your natural baseline.
You think your features are successful when they’re just growing with the business.
We help you establish baselines for your key metrics so you can measure true incremental impact.
It’s less glamorous than claiming “100 conversions from our campaign,” but it’s more honest and useful for making decisions.