Controlled DiD Experiments
Measure true causal SEO lift using synthetic holdout control cohorts vs treatment variants.
In organic search, user-level A/B testing cannot work: search engine spiders crawl URLs, not browser sessions. Simple "before vs. after" metrics are vulnerable to seasonal traffic swings and Google core updates.
HubSEO implements Controlled Difference-in-Differences (DiD) Experiments to isolate genuine search impact.
The Holdout Control Method
When an intervention targets multiple pages (such as title tag optimizations across 30 blog posts or schema additions on product templates), URLs are deterministically partitioned:
- Holdout Control Group (10%–20%): Left completely untouched as the reference baseline.
- Treatment Variant Group (80%–90%): Receives the proposed code or metadata change.
Control Cohort (Untouched) ───► Baseline Trend (Seasonality / Core Update)
Treatment Cohort (Modified) ───► Baseline Trend + True Causal SEO Lift
───────────────────────────────────────
Net Difference = True ImpactDifference-in-Differences Mathematics
Net causal uplift is computed by subtracting control baseline drift from treatment growth:
Net Uplift = ((T_post - T_pre) / T_pre) - ((C_post - C_pre) / C_pre)Statistical Significance Gate
Every active experiment is evaluated using a two-proportion Z-test over pre- and post-period conversion or click rates:
- Requires p < 0.05 (95% statistical confidence).
- Requires net causal uplift of at least 2.0%.
Decision Outcomes
| Decision | Meaning | Action |
|---|---|---|
rollout | Statistically significant positive lift ($p < 0.05, \Delta \ge +2%$) | Safe to deploy the fix across 100% of remaining holdout URLs. |
rollback | Statistically significant negative impact ($p < 0.05, \Delta \le -2%$) | Triggers instant automated rollback plan to restore previous HTML state. |
inconclusive | Result within natural noise bounds ($p \ge 0.05$) | Continue measurement window (up to 28 days) or expand sample size. |
Workspace Integration
The Experiments Panel lives in the unified project workspace at /dashboard/projects/[siteId]#experiments.
Pro, Business, and Admin tiers can monitor cohort sizes, launch live evaluations, and inspect net uplift curves.