A/B Test (Online Controlled Experiment)
Also known as: split test, controlled experiment, two-variant test, A/B Testi (Online Kontrollü Deney)
An A/B test is a randomized controlled experiment that simultaneously exposes two groups of users to a control variant (A) and a treatment variant (B) in order to determine whether a measured outcome differs significantly between them. The modern online controlled experiment framework was systematized by Ron Kohavi and colleagues at Microsoft in the early 2000s, building on R. A. Fisher's classical randomization principles from 1935. It is the dominant causal inference tool in web product development, digital marketing, and experimentation platforms.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use an A/B test when you can randomly assign users or units to exactly two variants and you have a clear primary metric. Key assumptions: users are assigned independently and at random (no leakage between groups); the experiment runs long enough to capture weekly seasonality; the sample ratio between groups matches the intended split (verify with a sample ratio mismatch check); and multiple-metric testing is corrected via Bonferroni or FDR procedures. A minimum of roughly 100 observations per group is needed for binary outcomes with moderate effect sizes; power analysis should confirm the required sample size before launch.
Strengths & limitations
- Provides strong causal evidence because randomization controls for all confounding variables simultaneously.
- Straightforward to interpret: a statistically significant result directly attributes outcome differences to the tested change.
- Flexible enough to cover binary outcomes (conversion rate), continuous outcomes (revenue, latency), and count outcomes (pages per session).
- Bayesian variants yield probabilistic statements (P(B > A)) that are often more actionable than a binary reject/fail-to-reject decision.
- Requires large sample sizes to detect small effects; underpowered experiments produce unreliable results.
- Cannot be used when random assignment is impossible or unethical.
- Network effects and cross-group interference (e.g. social platforms, marketplaces) violate the independence assumption and bias estimates.
- Running many simultaneous tests inflates the family-wise error rate unless multiple-comparison corrections are applied.
- SPRT and other continuous monitoring approaches are needed when peeking at results before the pre-planned end date; naive peeking inflates Type I error.
Frequently asked
How large does my sample need to be?
Run a power analysis before the experiment: specify the baseline conversion rate, the minimum detectable lift you care about, the desired power (usually 0.80), and the significance level (usually 0.05). The formula for a two-proportion z-test will give you the required n per group. As a rough rule of thumb, detecting a 10% relative lift on a 5% baseline requires roughly 5,000 observations per group.
Frequentist or Bayesian — which approach should I use?
Frequentist tests (z-test, t-test) control the Type I error rate at a specified level and are the regulatory default. Bayesian tests compute P(B > A) and allow optional stopping rules, making them popular in product teams that need to ship quickly. Both are valid; the choice depends on whether error-rate guarantees or probabilistic statements are more useful to your stakeholders.
What is a sample ratio mismatch and why does it matter?
A sample ratio mismatch (SRM) occurs when the proportion of users assigned to each variant differs from the intended ratio. Even small deviations — a 50/50 split landing at 48/52 — indicate a broken randomization mechanism (e.g. bot filtering, caching, redirect bugs). An SRM invalidates the causal interpretation; always run a chi-square test on the assignment counts before analyzing outcomes.
Can I run multiple A/B tests simultaneously?
Yes, provided the tests are on orthogonal segments or the platform implements proper interleaving controls. However, running many tests on the same population inflates the family-wise error rate. Apply Bonferroni or Benjamini-Hochberg FDR correction when monitoring several primary metrics, and be cautious about interaction effects between simultaneous experiments.
Sources
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. ISBN: 9781108724265
- Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. KDD '13. link ↗
How to cite this page
ScholarGate. (2026, June 1). A/B Test (Online Controlled Experiment). ScholarGate. https://scholargate.app/en/experimental-design/ab-testing
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Chi-square testStatistics↔ compare
- Independent t-testStatistics↔ compare
- Multi-Armed BanditExperimental design↔ compare
- Randomized Controlled TrialExperimental design↔ compare