Risk of Bias Assessment

What the Framework Is and Why It Matters

Risk-of-bias assessment asks whether weaknesses in a study's design, conduct, or reporting systematically distort its results. Unlike random error, bias introduces a directional skew rooted in methodological flaws. When this step is skipped in systematic reviews, low-quality studies can receive unwarranted weight and the evidence synthesis becomes misleading. Cochrane guidelines treat this assessment as mandatory, and evidence-grading frameworks such as GRADE translate study-level bias judgments directly into the overall certainty rating.

Core Tools and Their Domains

For randomized controlled trials, Cochrane RoB 2 evaluates five domains: the randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selective reporting. Each domain receives a judgment of low, some concerns, or high risk of bias, and the overall judgment follows the worst domain. ROBINS-I, developed for non-randomized studies, covers confounding, participant selection, classification of interventions, deviations, missing data, measurement of outcomes, and selective reporting. Both tools reach judgments through pre-specified signaling questions, ensuring consistency across reviewers.

How It Is Applied in Practice

The assessment follows these steps: selecting the appropriate tool, having two independent reviewers answer signaling questions, assigning domain-level judgments, resolving disagreements through discussion or a third reviewer, and reporting results in a table or traffic-light plot. To reduce influence of outcome knowledge, reviewers are often blinded to results, although this is not always feasible. Judgments are then carried into meta-analyses or GRADE evidence profiles, and studies judged at high risk may be examined separately in sensitivity analyses.

Common Pitfalls and Misconceptions

A frequent mistake is using composite quality-scoring scales such as the Jadad scale instead of domain-based tools; total scores mask domain-specific problems and are not recommended by Cochrane. Another error is conflating risk of bias with overall study quality: a rigorously conducted study can still carry high risk in one critical domain. Some researchers focus solely on blinding while underestimating threats from selective outcome reporting or missing data. Finally, leaving domain judgments without written justification undermines transparency and prevents other researchers from verifying or replicating the assessment.

Key terms

Systematic Error (Bias)

A directional, repeatable deviation from the true value, distinct from random error.

RoB 2

Cochrane's five-domain risk-of-bias tool for randomized controlled trials.

ROBINS-I

A seven-domain risk-of-bias tool for non-randomized studies of interventions.

Selective Reporting

Bias from reporting only outcomes or analyses that show statistically favorable results.

GRADE

Evidence-certainty grading framework that incorporates study-level risk-of-bias judgments.