Adaptive A/B Test — Adaptive A/B Testing
Adaptive A/B Testing · Also known as: adaptive AB test, bandit A/B test, multi-armed bandit testing, online adaptive experiment
An Adaptive A/B test is an experimental design that dynamically reallocates traffic or participants toward better-performing variants during the experiment itself, rather than holding allocations fixed until the end. Drawing on multi-armed bandit algorithms such as Thompson Sampling or Upper Confidence Bound (UCB), it balances the exploration of uncertain variants with the exploitation of those already showing superior performance, typically yielding higher aggregate outcomes while still producing valid inferential conclusions.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Adaptive A/B testing is most valuable when (1) exposure to an inferior variant carries a meaningful cost — in healthcare, digital products with high traffic, or time-limited campaigns; (2) the experiment runs long enough for the allocation algorithm to meaningfully shift (typically hundreds to thousands of observations); and (3) outcomes are observed quickly relative to the experiment's duration. It is also well-suited when testing multiple variants simultaneously (multi-arm setting) and when the goal is to maximize aggregate outcomes during the trial, not only after it. Do not use adaptive A/B testing when: the outcome is delayed relative to the decision epoch (delayed feedback makes updates unreliable); when interpretability of allocation ratios is required for regulatory review (e.g., some clinical trial contexts); when the sample size is very small (fewer than ~200 total observations), as the algorithm has insufficient data to meaningfully adapt; or when a simple balanced design will suffice and operational complexity is not warranted.
Strengths & limitations
- Reduces cumulative exposure to inferior variants compared with fixed equal allocation, which matters when the outcome metric is morally or commercially important.
- Well-suited to multi-arm settings where many variants are tested simultaneously and fixed allocation would spread the sample too thin.
- Produces valid statistical inference when combined with pre-registered stopping rules, despite the non-fixed allocation.
- Naturally integrates with Bayesian updating, enabling transparent quantification of uncertainty and evidence at any point in the trial.
- Flexible to context — applicable in online experimentation, clinical trials (response-adaptive randomization), and behavioral field experiments.
- Requires near-real-time or low-latency outcome observation; delayed feedback (e.g., outcomes measured weeks after treatment) breaks the update cycle and undermines adaptation.
- Statistical inference is more complex than in fixed-allocation designs — standard t-tests are invalid; specialized sequential or Bayesian testing procedures are required.
- Adaptive allocation can create imbalanced groups, which reduces power for detecting small effects and complicates subgroup analyses.
- Implementation requires technical infrastructure for real-time data collection, posterior updating, and traffic routing — not readily available in standard survey or lab platforms.
Frequently asked
Is an adaptive A/B test statistically valid?
Yes, provided that the stopping rule and analysis method are pre-registered and match the adaptive design. Standard t-tests and fixed-sample confidence intervals are not valid after adaptive allocation. Valid alternatives include alpha-spending sequential tests (e.g., O'Brien–Fleming boundaries), Thompson Sampling with Bayesian credible intervals, or off-policy correction estimators such as inverse propensity weighting. The key discipline is committing to the stopping rule before data collection begins.
How is this different from a standard multi-arm experiment?
A standard multi-arm experiment assigns participants to variants in fixed proportions (often equally) throughout the trial and analyzes all data at the end. An adaptive A/B test continuously updates the allocation ratios during the trial based on accumulating outcome data, so better-performing arms receive more participants over time. The inferential machinery must account for this non-fixed allocation, whereas standard multi-arm analysis assumes fixed randomization.
What is Thompson Sampling and why is it popular?
Thompson Sampling is a Bayesian algorithm that at each decision step samples a performance estimate from each variant's posterior distribution and assigns the next unit to the variant with the highest sampled value. It is popular because it is simple to implement, naturally trades off exploration and exploitation, and has been shown theoretically and empirically to achieve near-optimal cumulative outcomes. For binary outcomes it uses conjugate Beta distributions, making the posterior update a one-line calculation.
When should I stick with a classic fixed-allocation A/B test?
Use a fixed design when: outcomes are substantially delayed (feedback lag exceeds the inter-decision interval); the sample is small and the algorithm cannot meaningfully adapt; regulatory requirements mandate balanced allocation (e.g., certain Phase III clinical trials); or the sole objective is precise effect estimation rather than maximizing outcomes during the trial. In these situations a balanced randomized controlled design provides cleaner inference with simpler analysis.
Can adaptive A/B testing be used for more than two variants?
Yes. The multi-armed bandit framework naturally extends to any number of arms (variants). With many variants, adaptive allocation is especially valuable because fixed equal allocation in a multi-arm setting spreads the sample very thin across poorly-performing arms. Algorithms such as Thompson Sampling, UCB, and Successive Rejects scale directly to multi-arm settings.
Sources
- Russo, D., Van Roy, B., Kazerouni, A., Osband, I., & Wen, Z. (2018). A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning, 11(1), 1–96. DOI: 10.1561/2200000070 ↗
- Offer-Westort, M., Coppock, A., & Green, D. P. (2021). Adaptive Experimental Design: Prospects and Applications in Political Science. American Journal of Political Science, 65(4), 826–844. DOI: 10.1111/ajps.12597 ↗
How to cite this page
ScholarGate. (2026, June 3). Adaptive A/B Testing. ScholarGate. https://scholargate.app/en/experimental-design/adaptive-ab-test
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- AB DesignExperimental design↔ compare
- Adaptive ExperimentExperimental design↔ compare
- Blocked A/B TestExperimental design↔ compare
- Factorial A/B TestExperimental design↔ compare
- Multi-arm experimentExperimental design↔ compare
- Randomized Controlled TrialExperimental design↔ compare