Online Controlled Experiment

The OEC is the metric, or weighted set of metrics, that defines success for the experiment and ideally reflects long-term business value rather than a short-term proxy. Kohavi, Tang, and Xu argue it is the single most important and most difficult design decision, because experimentation will relentlessly optimize whatever you measure. If you pick clicks, you may ship clickbait that boosts clicks while eroding trust, revenue, or retention. A good OEC is sensitive enough to move within a reasonable test, hard to game, and aligned with strategic goals, often paired with guardrail metrics that must not degrade. Getting the OEC right is what separates experiments that improve the business from experiments that merely produce statistically significant but harmful 'wins.'

Classical fixed-horizon statistical tests assume you decide the sample size in advance and analyze once. If you instead watch the p-value continuously and stop the moment it crosses the significance threshold, you give random noise many chances to fake a win, which dramatically inflates the false-positive rate — well above the nominal level. Kohavi and colleagues highlight this as one of the most common and damaging mistakes in web experimentation. The fixes are to commit to a planned sample size and duration, or to use methods designed for continuous monitoring such as sequential testing or always-valid confidence intervals, which control error rates under repeated looks. Either way, results must not be cherry-picked at the first favorable moment.

A Sample Ratio Mismatch (SRM) occurs when the actual proportion of users in each variant differs significantly from the intended split, for example a 50/50 design that arrives as 52/48 at large scale. Because the assignment is supposed to be random, a meaningful deviation signals a bug — a broken redirect, asymmetric logging, bot filtering, or a flawed randomization unit — that almost always biases the results, often masquerading as a lift. Kohavi's work treats the SRM test (a simple chi-square on the observed versus expected counts) as a mandatory trustworthiness gate: if it fails, the experiment's metrics are not to be trusted no matter how attractive the apparent effect, and the root cause must be fixed before any conclusion is drawn.