Stochastic Markov Model — Probabilistic State-Transition Simulation with Uncertainty Propagation
Also known as: Probabilistic Markov Model, Stochastic Markov Chain, SMM, Monte Carlo Markov Model
A Stochastic Markov Model is a simulation technique that represents a system as a set of mutually exclusive health or decision states, moves a cohort (or individual agents) through those states using probabilistically sampled transition parameters, and aggregates outcomes across thousands of Monte Carlo iterations to produce full probability distributions over costs, outcomes, or rankings rather than single point estimates.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+4 more
When to use it
Use a Stochastic Markov Model when decision-making involves discrete states, recurring transition events over multiple time periods, and parameter uncertainty that must be formally propagated to outputs — most commonly in health technology assessment, pharmacoeconomics, and policy evaluation where uncertainty in transition probabilities and payoffs is substantial. It is the standard approach when regulatory or HTA bodies (NICE, CADTH, etc.) require probabilistic sensitivity analysis. Do NOT use it when the system has continuous state spaces better handled by differential equations, when the Markov assumption (memorylessness) is implausible and history clearly matters without being capturable via state expansion, when the analysis horizon is a single period, or when data are too sparse to justify specifying parameter distributions (in that case a deterministic one-way sensitivity analysis may be more appropriate).
Strengths & limitations
- Explicitly quantifies parameter uncertainty and delivers credible intervals rather than point estimates, supporting transparent decision-making under uncertainty.
- The probabilistic output (e.g., cost-effectiveness acceptability curves) directly informs value-of-information analyses and budget allocation decisions.
- State-transition structure is interpretable to non-technical stakeholders, and model logic can be audited cycle by cycle.
- Naturally accommodates multiple simultaneously uncertain parameters without requiring manual scenario construction.
- Widely accepted by health technology assessment bodies (NICE, CADTH, PBAC) as the standard for probabilistic sensitivity analysis in economic evaluations.
- Extensible: individual-level microsimulation and time-inhomogeneous variants can incorporate patient heterogeneity and time-varying transitions within the same framework.
- The Markov (memorylessness) assumption — that future transitions depend only on the current state, not history — is often violated in practice; workarounds via tunnel states add model complexity.
- Specifying credible probability distributions for all parameters requires good-quality evidence; poorly specified priors propagate poorly calibrated uncertainty.
- Computational cost grows with the number of states, cycles, and Monte Carlo iterations; very large models may require significant runtime or parallel computing resources.
- Model structure itself (which states to include, cycle length) is typically fixed and not subject to probabilistic uncertainty, which can be a major source of unquantified structural uncertainty.
- Results are sensitive to the choice of distributions and their correlations; ignoring correlation between parameters (e.g., costs and utilities) can underestimate or overestimate total variance.
Frequently asked
What is the difference between a stochastic and a deterministic Markov model?
A deterministic Markov model uses fixed point estimates for all transition probabilities and payoffs, producing a single outcome value. A stochastic Markov model samples these parameters from probability distributions in each Monte Carlo iteration, producing a distribution of outcomes that reflects parameter uncertainty. Both use the same cohort-trace mechanics; only the parameter treatment differs.
How many Monte Carlo iterations are sufficient?
A minimum of 1000 iterations is commonly recommended; 10,000 is typical for stable credible intervals and cost-effectiveness acceptability curves. Convergence should be verified visually (running mean plots) or with formal diagnostics. Models with many correlated parameters or rare events may require more iterations.
When does the Markov assumption fail, and how can it be addressed?
The Markov assumption fails when the probability of transitioning out of a state depends on how long a patient has been in it (e.g., increasing mortality risk with disease duration). The standard fix is to introduce tunnel states that track time-in-state explicitly, at the cost of a larger state space. Alternatively, individual-level microsimulation can track each patient's history directly.
Can a stochastic Markov model handle individual heterogeneity?
Standard cohort-level stochastic Markov models capture only parameter uncertainty across iterations, not individual-level variability. To incorporate patient heterogeneity (different baseline risks, comorbidities), the model must be extended to an individual-level microsimulation where each simulated patient is tracked separately through the state space.
How is a stochastic Markov model different from a discrete-event simulation?
In a Markov model, time advances in fixed cycles and all events are governed by constant per-cycle transition probabilities. Discrete-event simulation models individual entities whose events occur at continuous times drawn from arbitrary distributions, allowing more flexible representation of queuing, resource constraints, and time-varying rates. Markov models are simpler to build and validate; DES is more flexible but harder to parameterise.
Sources
- Sonnenberg, F. A., & Beck, J. R. (1993). Markov models in medical decision making: A practical guide. Medical Decision Making, 13(4), 322–338. DOI: 10.1177/0272989X9301300409 ↗
- Briggs, A., Sculpher, M., & Claxton, K. (2006). Decision Modelling for Health Economic Evaluation. Oxford University Press. ISBN: 9780198526629
How to cite this page
ScholarGate. (2026, June 3). Stochastic Markov Model — Probabilistic State-Transition Simulation with Uncertainty Propagation. ScholarGate. https://scholargate.app/en/simulation/stochastic-markov-model
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Discrete-Event SimulationSimulation↔ compare
- Markov ModelSimulation↔ compare
- MicrosimulationSimulation↔ compare
- MONTE-CARLO-SIMULATIONDecision-making↔ compare
- SENSITIVITY-ANALYSISDecision-making↔ compare
- Stochastic Dynamic ProgrammingSimulation↔ compare