Factorial A/B Test — Multi-Factor Online Controlled Experiment
Factorial A/B Test (Multi-Factor Online Experiment) · Also known as: factorial split test, multi-factor A/B test, factorial online experiment, factorial controlled experiment
A factorial A/B test is a controlled online experiment that simultaneously manipulates two or more independent factors, each at two or more levels, exposing different user groups to every combination of factor levels. Rooted in Fisher's factorial design and operationalised at scale by tech companies, it enables researchers to estimate both the independent main effect of each factor and the interaction effects between factors — all from a single experimental run.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use a factorial A/B test when you need to evaluate two or more independent changes in a single experiment and when you have reason to suspect those factors might interact. It is the right choice when traffic is sufficient to power all cells simultaneously — a rough rule is that you need the same per-cell sample as a standard A/B test, multiplied by the number of cells. Do not use it when traffic is scarce and you cannot power all cells; in that case, run sequential A/B tests instead. Avoid it when factors are known to be fully independent and interaction effects are theoretically implausible — in that case, separate A/B tests are simpler to communicate. Also avoid it when more than four or five factors are involved without a fractional factorial strategy, as the number of cells and the complexity of interpretation grow combinatorially.
Strengths & limitations
- Tests multiple factors in a single experiment, reducing total calendar time and traffic cost compared to sequential A/B tests.
- Detects and quantifies interaction effects that sequential testing cannot reveal, preventing the deployment of combinations whose benefits do not stack.
- Full factorial designs are statistically efficient — orthogonal factor assignment means main effects and interactions are estimated independently.
- Supports clear, pre-specified analysis plans and is compatible with standard frequentist (ANOVA, regression) and Bayesian inference frameworks.
- Widely adopted in industry (tech, e-commerce, pharma) and backed by a mature literature on design and analysis.
- Sample-size requirements scale with the number of cells; a 2×2×2 design needs eight times the per-cell N of a simple two-arm test, which may be infeasible for low-traffic products.
- High-order interactions (three-way, four-way) are difficult to interpret and often underpowered even in large experiments.
- Results are specific to the user population and context of the experiment and may not generalize across different platforms, seasons, or markets.
- Analysis and communication of interaction effects require statistical sophistication that stakeholders may not have.
Frequently asked
How is a factorial A/B test different from running multiple separate A/B tests?
Sequential A/B tests cannot detect whether two factors interact — you might find that each change improves conversion individually but that deploying both actually hurts performance. A factorial A/B test runs all combinations simultaneously, so it estimates interaction effects directly. It also finishes faster if you have adequate traffic, since you test everything in one experiment rather than in sequence.
When should I use a fractional rather than a full factorial design?
A full factorial tests every combination of every factor level. When the number of factors is large (typically five or more), the number of cells becomes impractically large. A fractional factorial design tests only a strategically chosen subset of cells — enough to estimate main effects and low-order interactions while sacrificing information about high-order interactions that are rarely practically important. If you have many factors and limited traffic, consider a fractional factorial or a sequential testing strategy.
How do I handle multiple comparisons in a factorial A/B test?
Each main effect and each interaction term is a separate hypothesis. Without adjustment, the familywise false-discovery rate increases with the number of tests. Common approaches include Bonferroni correction (conservative, controls familywise error), Benjamini-Hochberg procedure (controls false discovery rate, more power), or a Bayesian framework that naturally propagates uncertainty across all effects. The correction method should be pre-specified in the analysis plan.
What sample size do I need?
A standard rule is to power each cell as you would a two-arm A/B test — compute the required N for your expected effect size and desired power, then multiply by the total number of cells. For a 2×2 design that means four times the per-arm N of a standard A/B test. If powering all cells is infeasible, consider a fractional factorial design or reducing the number of factors.
Can I analyze a factorial A/B test with a simple t-test per pair?
No. Pairwise t-tests between cells ignore the factorial structure, inflate the false-discovery rate, and — most critically — do not estimate interaction effects. The correct analysis is a factorial ANOVA or an equivalent regression model with indicator variables for each factor and their products (interaction terms). Only this approach correctly partitions the variance into main effects and interactions.
Sources
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. ISBN: 978-1108724265
- Montgomery, D. C. (2017). Design and Analysis of Experiments (9th ed.). Wiley. ISBN: 978-1119113478
How to cite this page
ScholarGate. (2026, June 3). Factorial A/B Test (Multi-Factor Online Experiment). ScholarGate. https://scholargate.app/en/experimental-design/factorial-ab-test
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- AB DesignExperimental design↔ compare
- Adaptive A/B testExperimental design↔ compare
- Factorial ExperimentExperimental design↔ compare
- Fractional Factorial ExperimentExperimental design↔ compare
- Full Factorial ExperimentExperimental design↔ compare
- Multi-arm experimentExperimental design↔ compare