Pilot Natural Experiment — Feasibility Testing Before a Full Natural Experiment

The design logic is identical — both exploit exogenous variation to approximate causal inference. The difference is scope and purpose. A pilot is conducted on a small subset of the eventual study population specifically to assess feasibility: data quality, group comparability, and rough effect size. Its findings are used to refine or abandon the plan for the full study, not to draw causal conclusions.

Pilot findings can be published, but they should be framed honestly as preliminary and underpowered. The appropriate contribution is methodological — demonstrating that the design is feasible and reporting the effect-size estimate used to power the full study — rather than claiming definitive causal estimates.

The pilot itself does not typically require preregistration, but the full study should be preregistered before the full dataset is analysed, using the analysis plan informed by the pilot. Preregistering after seeing pilot results and before seeing full results is acceptable and improves credibility.

There is no universal rule. The pilot must be large enough to provide meaningful preliminary estimates of effect size, variance, and data quality — typically 10–30% of the anticipated full-study sample is cited as a rough guide in the pilot study literature. In natural experiments, the sample size is often constrained by the number of naturally treated units, so the pilot may simply use whatever data are accessible early.

This is a success, not a failure. Discovering that the exogenous variation is endogenous, that groups are not comparable, or that data are inaccessible before committing to a full study saves substantial resources and prevents publishing invalid causal estimates. The researcher should then consider alternative designs — randomized experiments if feasible, or other quasi-experimental approaches.