Evaluation-Oriented Multilevel Mixed Methods Design
Also known as: multilevel mixed methods evaluation, hierarchical mixed methods evaluation, MLM mixed methods, nested mixed methods evaluation
Evaluation-oriented multilevel mixed methods is a research design that combines quantitative and qualitative data across hierarchically nested levels of an organization or system — such as students within classrooms within schools — to evaluate a program, policy, or intervention. By capturing outcomes, processes, and contextual factors simultaneously at each level, this design produces richer evaluative inferences than either purely statistical multilevel models or single-level qualitative evaluations alone.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use this design when evaluating programs or policies operating within hierarchically organized systems — education, health, social services, government — where outcomes at one level are embedded within and shaped by conditions at a higher level. It is especially valuable when stakeholders need both accountability evidence (effect sizes, cost-benefit) and explanatory understanding (why effects occurred or varied). Do not use it when the evaluation context is genuinely flat (no meaningful nesting), when resources do not allow data collection at multiple levels, or when qualitative depth is needed without any quantitative measurement.
Strengths & limitations
- Captures both the magnitude and the mechanisms of program effects across organizational levels.
- Addresses the fundamental mismatch between where interventions are delivered and where outcomes are measured.
- Produces evaluation inferences that are credible to quantitatively-oriented funders and contextually meaningful to practitioners.
- Supports equity analysis by examining whether effects differ across subgroups or sites.
- Integrates stakeholder perspectives at each level through the qualitative strand.
- Requires substantially more resources, time, and expertise than single-strand or single-level evaluations.
- Demands competency in both multilevel quantitative modeling and rigorous qualitative methods — rarely found in a single evaluator.
- Integration of strands across levels adds logical complexity; inferences can be difficult to communicate to non-technical audiences.
- Minimum sample sizes at higher levels (e.g., at least 20 schools) are often unattainable in small-scale program evaluations.
Frequently asked
Is this the same as a multilevel model (HLM) with some qualitative data added on?
No. Adding a few interviews to an HLM study is not the same as an evaluation-oriented multilevel mixed methods design. The defining feature is deliberate, theorized integration: the qualitative strand is planned and executed at each relevant level, and its findings are formally mixed with the quantitative results at a specified mixing point to generate joint meta-inferences. The design also frames the entire enterprise within an evaluation logic — assessing merit, worth, and significance of a program.
How many levels are typical?
Most applications involve two to three levels — for example, students within schools, or patients within clinics within hospitals. Beyond three levels the design becomes logistically and analytically demanding. The number of levels should be determined by the theoretical model of the program, not by convenience.
What does 'mixing' actually mean in this design?
Mixing is the active integration of quantitative and qualitative data or findings. Common mixing strategies include joint displays (tables that show quantitative outcomes alongside qualitative themes side by side), using statistical outliers (unusually high or low performing sites) to select cases for qualitative follow-up, and building a merged data matrix where quantitative scores and qualitative codes are combined for each unit.
Do I need an HLM specialist on my team?
For evaluations with genuine nesting and an ICC above a trivial threshold, a researcher with multilevel modeling competency is strongly advisable. Ignoring clustering or using ordinary regression when intraclass correlation is non-trivial produces anticonservative significance tests. If HLM expertise is unavailable, cluster-robust standard errors or design-effect corrections are minimum acceptable alternatives.
Can this design be used with a transformative or participatory evaluation framework?
Yes. Mertens and others have explicitly argued for embedding mixed methods within transformative and participatory frameworks, particularly when the evaluation concerns marginalized or underserved communities. The multilevel structure is compatible with social justice orientations, as it can spotlight inequities across levels and incorporate community voice through the qualitative strand.
Sources
- Mertens, D. M. (2010). Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods (3rd ed.). Sage. ISBN: 978-1412975551
- Creswell, J. W., & Plano Clark, V. L. (2018). Designing and Conducting Mixed Methods Research (3rd ed.). Sage. ISBN: 978-1483358468
How to cite this page
ScholarGate. (2026, June 3). Evaluation-Oriented Multilevel Mixed Methods Design. ScholarGate. https://scholargate.app/en/research-design/evaluation-oriented-multilevel-mixed-methods
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Multilevel ModelingResearch Statistics↔ compare
- Program EvaluationField Methods↔ compare