Policy Evaluation Causal Impact Analysis
Also known as: policy causal impact, BSTS policy evaluation, Bayesian policy impact assessment, CIA policy evaluation
Policy Evaluation Causal Impact Analysis applies the Bayesian structural time-series (BSTS) framework of Brodersen et al. (2015) to estimate the causal effect of a policy intervention on aggregate outcomes. By constructing a synthetic counterfactual from pre-policy data and control covariates, it asks: what would have happened had the policy not been enacted? The difference between observed and predicted post-policy outcomes is the estimated policy effect.
Key highlights
- Constructs a data-driven counterfactual without requiring a hand-picked control group, using Bayesian variable selection across many candidate covariates.
- Produces full posterior distributions for both pointwise and cumulative impact, providing honest credible intervals rather than overconfident p-values.
- Handles complex seasonal and trend dynamics inherent in aggregate policy data through the structural time-series decomposition.
- Yields immediately interpretable outputs — observed vs. predicted plots and cumulative impact summaries — that are accessible to policy audiences.
- Accommodates aggregate-level data where unit-level randomization is impossible, making it practical for real-world policy contexts.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use Policy Evaluation Causal Impact Analysis when: (1) a policy or regulation was rolled out at a known date to a whole unit (country, state, city, sector); (2) outcome data are available as a time series with a sufficient pre-policy baseline (typically at least 30–50 pre-period observations); and (3) at least a few co-varying control time series exist that were not affected by the policy and correlate with the outcome in the pre-period. Do not use it when the pre-policy series is very short (fewer than ~20 observations), when no valid control covariates exist, when the intervention was gradual and its start date is ambiguous, or when the outcome is cross-sectional rather than temporal.
Strengths & limitations
- Constructs a data-driven counterfactual without requiring a hand-picked control group, using Bayesian variable selection across many candidate covariates.
- Produces full posterior distributions for both pointwise and cumulative impact, providing honest credible intervals rather than overconfident p-values.
- Handles complex seasonal and trend dynamics inherent in aggregate policy data through the structural time-series decomposition.
- Yields immediately interpretable outputs — observed vs. predicted plots and cumulative impact summaries — that are accessible to policy audiences.
- Accommodates aggregate-level data where unit-level randomization is impossible, making it practical for real-world policy contexts.
- Requires valid control covariates that track the outcome pre-policy but are unaffected post-policy; if no such series exists, the counterfactual is poorly identified.
- A short pre-policy baseline limits the model's ability to capture seasonality and trends, degrading counterfactual accuracy.
- Cannot disentangle simultaneous policy changes or confounding events that coincide with the intervention — the estimated effect may conflate multiple causes.
- The Bayesian BSTS framework is computationally intensive and requires familiarity with model diagnostics beyond standard regression output.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How does this differ from standard Interrupted Time Series analysis?
Standard interrupted time series models a single outcome series and estimates a level or slope change at the intervention point. Causal Impact additionally uses control covariates — related time series unaffected by the policy — to construct a richer synthetic counterfactual, often improving precision when the pre-period is moderately short or volatile.
How many pre-policy observations do I need?
A practical minimum is around 30–50 observations to identify trend and seasonal components reliably. Fewer observations leave the BSTS model poorly constrained, widening credible intervals to the point of limited utility.
What makes a valid control covariate?
A control series should (1) correlate with the outcome during the pre-policy period, (2) not have been affected by the policy itself, and (3) not be causally downstream of the outcome. Examples include the same outcome measured in an unaffected region, or a closely related aggregate indicator unconnected to the intervention.
Can I use this method if the policy was rolled out gradually?
Gradual rollouts are problematic because the intervention date is ambiguous, which undermines the sharp pre/post split the model requires. In such cases, interrupted time series with a dose-response specification or difference-in-differences exploiting variation in rollout timing across units is typically more appropriate.
How do I handle a concurrent event that coincides with the policy?
A concurrent shock that affected only the treated unit and not the control covariates cannot be separated from the policy effect — the estimate conflates both. Document the confounding event explicitly, report sensitivity analyses over different post-intervention windows, and temper causal language accordingly.
Sources
- 1.Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015). Inferring causal impact using Bayesian structural time-series models. Annals of Applied Statistics, 9(1), 247-274.
- 2.Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493-505.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Policy Evaluation Causal Impact Analysis. ScholarGate. https://scholargate.app/causal-inference/policy-evaluation-causal-impact-analysis