Policy Evaluation Counterfactual Impact Evaluation (CIE)
Counterfactual Impact Evaluation for Policy Assessment · Also known as: CIE, policy CIE, counterfactual policy evaluation, impact evaluation
Counterfactual Impact Evaluation (CIE) for policy assessment estimates the causal effect of a public policy or programme by comparing observed outcomes of participants against a rigorously constructed counterfactual — what would have happened had the policy not existed. Rooted in the Rubin potential-outcomes framework, CIE is the standard methodology endorsed by the European Commission for evaluating research, innovation, and structural funding programmes.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use Policy Evaluation CIE when a clearly defined public programme or policy has been implemented and you need a credible causal estimate of its impact for accountability, learning, or resource-allocation decisions. The method is most appropriate when randomised assignment was not feasible, administrative or survey data link participants to comparable non-participants, and at least one quasi-experimental lever is available (a threshold, a staggered rollout, an instrument, or panel data with pre-treatment observations). It is not appropriate when the comparison group is entirely non-comparable, when policy exposure is universal (no untreated units exist), when outcomes are measured only cross-sectionally without a pre-treatment baseline, or when the mechanism of interest requires structural rather than reduced-form identification.
Strengths & limitations
- Provides a credible causal estimate of programme effects without requiring randomisation, making it feasible for real-world policy settings.
- Accommodates multiple estimation strategies (matching, DiD, RDD, IV), allowing the analyst to exploit whatever quasi-experimental variation is present in the data.
- Endorsed by major evaluation bodies (European Commission, World Bank, OECD), ensuring policy uptake and cross-study comparability.
- Triangulating across methods strengthens credibility when no single approach is fully defensible.
- Can handle heterogeneous treatment effects by estimating impacts for relevant subgroups (firm size, sector, region).
- Each underlying estimator carries its own untestable or partially testable assumption (unconfoundedness, parallel trends, instrument validity); if these fail, the causal interpretation is invalid.
- Data requirements are demanding: linked administrative records or panel surveys with pre-treatment observations are typically needed.
- Selection bias in programme participation is a persistent challenge; sophisticated matching or quasi-experimental designs may not fully resolve it.
- Effect estimates are typically local (local ATT or LATE), limiting generalisability to the evaluated programme and context.
- Implementation is resource-intensive, requiring substantial methodological expertise and high-quality data infrastructure.
Frequently asked
What distinguishes policy CIE from a standard impact evaluation?
Policy CIE refers specifically to the structured framework — endorsed by institutions such as the European Commission — that applies counterfactual methods (matching, DiD, RDD, IV) to public programme assessment, with explicit attention to additionality, comparison group validity, and multi-method robustness. Standard impact evaluation is the broader umbrella term; policy CIE is a formalised, institutionally recognised implementation of it.
Which counterfactual method should I choose?
The choice depends on how assignment to the programme occurred. If eligibility followed a numerical threshold, RDD is natural. If the programme rolled out at different times across units, staggered DiD or event-study designs are appropriate. If participation was voluntary but administrative data on non-participants exist, matching or IPW on observables is common. IV is reserved for cases where a credible instrument for participation can be identified. When multiple methods are feasible, apply all and report consistency.
What is 'additionality' and how does CIE measure it?
Additionality is the portion of an outcome that would not have occurred without the policy — i.e., the true causal effect. CIE measures it as the ATT: the gap between participants' observed outcomes and their counterfactual outcomes under non-participation. Positive additionality means the programme caused a genuine improvement beyond what participants would have achieved on their own.
How do I handle a universal programme with no untreated units?
Universal coverage removes the comparison group, making standard CIE infeasible. Alternatives include comparing outcomes before and after rollout (interrupted time series), using geographic or administrative units that adopted the programme at different times (staggered rollout DiD), or constructing a synthetic control from a set of similar but untreated regions or countries.
How large a sample is needed for reliable CIE estimates?
There is no universal threshold, but the effective sample of matched pairs or treated units governs precision. For matching-based CIE, at least several hundred treated units are typically needed to achieve stable propensity score estimates and adequate post-matching balance. For RDD, the precision depends on the bandwidth and density around the threshold. Always report power calculations or minimum detectable effects when planning a prospective evaluation.
Sources
- Imbens, G. W., & Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press. ISBN: 978-0521885881
- Cerulli, G. (2014). Econometric Evaluation of Socioeconomic Programs: Theory and Applications. Springer. link ↗
How to cite this page
ScholarGate. (2026, June 3). Counterfactual Impact Evaluation for Policy Assessment. ScholarGate. https://scholargate.app/en/causal-inference/policy-evaluation-counterfactual-impact-evaluation
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Counterfactual Impact EvaluationCausal inference↔ compare
- Difference-in-DifferencesEconometrics↔ compare
- Instrumental Variables in Health ResearchHealth Economics↔ compare
- Propensity Score MatchingResearch Statistics↔ compare
- Synthetic Control MethodCausal inference↔ compare