Blocked A/B Test — Block-Randomized Split Test
Blocked A/B Test (Block-Randomized Split Test) · Also known as: block-randomized A/B test, stratified A/B test, blocked split test, block-design A/B experiment
A blocked A/B test is an experimental design that partitions units (users, subjects, or clusters) into homogeneous blocks before randomly assigning them to treatment A or treatment B within each block. Blocking reduces within-experiment noise by ensuring that known sources of variation — such as device type, geography, or user tenure — are balanced across conditions, yielding more precise estimates of the treatment effect than a simple unblocked A/B test.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use a blocked A/B test when you know in advance that a covariate (device type, user segment, geographic region, time period) is correlated with your primary metric and you want to prevent imbalance on that covariate from inflating your error estimate. It is especially valuable when sample sizes are moderate and you cannot afford the variance that simple randomization would leave in. Do not use blocking when you have no meaningful prognostic covariates — the overhead is unnecessary and block-size constraints may reduce power. Avoid it when the blocking variable is too fine-grained to yield blocks with at least a few units per condition, or when block membership cannot be determined before assignment.
Strengths & limitations
- Reduces experimental error variance attributable to known prognostic covariates, increasing statistical power without increasing sample size.
- Guards against accidental covariate imbalance that could bias treatment effect estimates in finite samples.
- Block estimates can themselves provide useful insights into heterogeneous treatment effects across segments.
- Widely supported in standard statistical packages (ANOVA, regression with indicators, mixed models).
- Directly analogous to the gold-standard block design used in clinical and agricultural research, giving the approach strong methodological credibility.
- Requires pre-identification of blocking variables; variables not known or not measured before assignment cannot be used for blocking.
- Very small block sizes (e.g., only 2–4 units per block) reduce the reliability of within-block randomization and can harm power rather than help it.
- Analysis is more complex than a simple two-sample t-test; the block structure must be correctly incorporated in the statistical model.
- Blocking on the wrong variable (one uncorrelated with the outcome) provides no variance reduction and adds analytical overhead.
Frequently asked
What is the difference between a blocked A/B test and a stratified A/B test?
The terms are often used interchangeably in applied settings. Strictly, blocking is a design concept (assignment of units within groups) while stratification can refer to either design or analysis (post-hoc re-weighting). In a blocked A/B test, randomization happens separately within each block before exposure. In a stratified analysis, you may have randomized without blocking but reweight or adjust post-hoc by stratum. Blocking at design time is generally more powerful than stratification at analysis time.
How do I choose which variables to block on?
Block on variables that (1) you can measure before assigning treatment, (2) are expected to correlate with the primary outcome metric, and (3) are not caused by the treatment itself. Common choices in online experiments include device type, user tenure cohort, and geographic region. Avoid blocking on variables with many categories unless your sample is large enough to maintain adequate block sizes.
Does blocking change the sample size requirement?
Blocking can reduce the sample size needed to achieve a given power level, because it removes variance attributable to the blocking variable from the error term. The efficiency gain depends on how strongly the blocking variable correlates with the outcome. If the correlation is low, blocking yields little benefit and a simple A/B test may require fewer analytical resources.
Can I use a blocked design with more than two variants?
Yes. When there are more than two conditions the design is called a blocked multi-arm experiment or a randomized complete block design with multiple treatments. Units within each block are randomly assigned to one of k conditions. The analysis generalizes to ANOVA or regression with block and treatment indicators, followed by planned or post-hoc comparisons.
What happens if I forget to include block indicators in my analysis?
Omitting block indicators means the variance reduction from blocking is lost. Worse, if block sizes are unequal, the naive estimate may be biased. Always include block fixed effects (or a random-effects term) in the statistical model to honor the blocked design and recover the efficiency gains.
Sources
- Fisher, R. A. (1926). The arrangement of field experiments. Journal of the Ministry of Agriculture of Great Britain, 33, 503–513. link ↗
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. ISBN: 9781108724265
How to cite this page
ScholarGate. (2026, June 3). Blocked A/B Test (Block-Randomized Split Test). ScholarGate. https://scholargate.app/en/experimental-design/blocked-ab-test
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Blocked Randomized Controlled TrialExperimental design↔ compare
- Factorial A/B TestExperimental design↔ compare
- Multi-arm experimentExperimental design↔ compare
- Randomized Controlled TrialExperimental design↔ compare