Policy Evaluation Causal Impact Analysis
Policy Evaluation Causal Impact Analysis via Bayesian Structural Time-Series · Also known as: policy causal impact, BSTS policy evaluation, Bayesian policy impact assessment, CIA policy evaluation
Policy Evaluation Causal Impact Analysis applies the Bayesian structural time-series (BSTS) framework of Brodersen et al. (2015) to estimate the causal effect of a policy intervention on aggregate outcomes. By constructing a synthetic counterfactual from pre-policy data and control covariates, it asks: what would have happened had the policy not been enacted? The difference between observed and predicted post-policy outcomes is the estimated policy effect.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use Policy Evaluation Causal Impact Analysis when: (1) a policy or regulation was rolled out at a known date to a whole unit (country, state, city, sector); (2) outcome data are available as a time series with a sufficient pre-policy baseline (typically at least 30–50 pre-period observations); and (3) at least a few co-varying control time series exist that were not affected by the policy and correlate with the outcome in the pre-period. Do not use it when the pre-policy series is very short (fewer than ~20 observations), when no valid control covariates exist, when the intervention was gradual and its start date is ambiguous, or when the outcome is cross-sectional rather than temporal.
Strengths & limitations
- Constructs a data-driven counterfactual without requiring a hand-picked control group, using Bayesian variable selection across many candidate covariates.
- Produces full posterior distributions for both pointwise and cumulative impact, providing honest credible intervals rather than overconfident p-values.
- Handles complex seasonal and trend dynamics inherent in aggregate policy data through the structural time-series decomposition.
- Yields immediately interpretable outputs — observed vs. predicted plots and cumulative impact summaries — that are accessible to policy audiences.
- Accommodates aggregate-level data where unit-level randomization is impossible, making it practical for real-world policy contexts.
- Requires valid control covariates that track the outcome pre-policy but are unaffected post-policy; if no such series exists, the counterfactual is poorly identified.
- A short pre-policy baseline limits the model's ability to capture seasonality and trends, degrading counterfactual accuracy.
- Cannot disentangle simultaneous policy changes or confounding events that coincide with the intervention — the estimated effect may conflate multiple causes.
- The Bayesian BSTS framework is computationally intensive and requires familiarity with model diagnostics beyond standard regression output.
Frequently asked
How does this differ from standard Interrupted Time Series analysis?
Standard interrupted time series models a single outcome series and estimates a level or slope change at the intervention point. Causal Impact additionally uses control covariates — related time series unaffected by the policy — to construct a richer synthetic counterfactual, often improving precision when the pre-period is moderately short or volatile.
How many pre-policy observations do I need?
A practical minimum is around 30–50 observations to identify trend and seasonal components reliably. Fewer observations leave the BSTS model poorly constrained, widening credible intervals to the point of limited utility.
What makes a valid control covariate?
A control series should (1) correlate with the outcome during the pre-policy period, (2) not have been affected by the policy itself, and (3) not be causally downstream of the outcome. Examples include the same outcome measured in an unaffected region, or a closely related aggregate indicator unconnected to the intervention.
Can I use this method if the policy was rolled out gradually?
Gradual rollouts are problematic because the intervention date is ambiguous, which undermines the sharp pre/post split the model requires. In such cases, interrupted time series with a dose-response specification or difference-in-differences exploiting variation in rollout timing across units is typically more appropriate.
How do I handle a concurrent event that coincides with the policy?
A concurrent shock that affected only the treated unit and not the control covariates cannot be separated from the policy effect — the estimate conflates both. Document the confounding event explicitly, report sensitivity analyses over different post-intervention windows, and temper causal language accordingly.
Sources
- Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015). Inferring causal impact using Bayesian structural time-series models. Annals of Applied Statistics, 9(1), 247-274. DOI: 10.1214/14-AOAS788 ↗
- Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493-505. DOI: 10.1198/jasa.2009.ap08746 ↗
How to cite this page
ScholarGate. (2026, June 3). Policy Evaluation Causal Impact Analysis via Bayesian Structural Time-Series. ScholarGate. https://scholargate.app/en/causal-inference/policy-evaluation-causal-impact-analysis
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Bayesian Causal Impact AnalysisCausal inference↔ compare
- Causal Impact AnalysisCausal inference↔ compare
- Difference-in-DifferencesEconometrics↔ compare
- Interrupted Time SeriesCausal inference↔ compare
- Policy Evaluation Interrupted Time SeriesCausal inference↔ compare
- Synthetic Control MethodCausal inference↔ compare