Kalman Filter with Missing Data
Kalman Filter for State-Space Models with Missing Observations · Also known as: Kalman smoother with missing data, state-space model with missing observations, EM Kalman filter, KF-EM
The Kalman filter with missing data extends the classical Kalman filter to handle time series in which some observations are absent. When an observation is missing at time t the update step is skipped and the state estimate is carried forward from the prediction step alone. Combined with the Expectation-Maximisation (EM) algorithm, the approach also estimates unknown model parameters from incomplete data, making it a practical tool for real-world irregularly observed series.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use the Kalman filter with missing data for multivariate or univariate time series that are observed at regular intervals but have intermittent gaps — sensor outages, non-trading days in financial data, irregular survey waves, or sporadic lab measurements. It is particularly well-suited when the underlying dynamics are linear and Gaussian, and when you want parameter estimates and smoothed state trajectories alongside imputed values. Avoid it when missingness is heavy and informative (data missing because the system failed), when dynamics are strongly non-linear (use particle filters or unscented Kalman filter instead), or when the series is very short relative to the number of parameters to be estimated via EM.
Strengths & limitations
- Handles missing observations naturally within the filter recursion — no separate imputation required.
- Produces smoothed state estimates and imputed values for missing time points with associated uncertainty.
- The EM variant simultaneously estimates model parameters and missing data from a single likelihood.
- Scales efficiently to multivariate systems with many observed series.
- Provides the exact likelihood for the observed data, enabling principled model comparison via AIC or BIC.
- Assumes linear Gaussian state-space dynamics; strongly non-linear or non-Gaussian systems require alternatives.
- EM convergence can be slow and may reach a local rather than global maximum of the likelihood.
- With many missing observations and many unknown parameters, the EM M-step may become ill-conditioned.
- Informative missingness (data absent for system-related reasons) violates the missing-at-random assumption.
Frequently asked
What exactly happens in the filter when an observation is missing?
The predict step runs as usual, but the update step is skipped: the Kalman gain is set to zero, so the filtered state equals the predicted state and the filtered covariance equals the predicted covariance. Uncertainty is not reduced for that time point because no new information arrived.
Do I need EM if I know the model parameters?
No. If F, H, Q, R and the initial state distribution are fully specified, you can run the Kalman filter and smoother directly with the missing-observation rule. EM is only needed when those parameters must be estimated from the data.
When should I use a particle filter instead?
If the state-space dynamics are non-linear or the noise distributions are non-Gaussian, the linear Kalman equations are no longer exact. In those cases a particle filter (sequential Monte Carlo) approximates the non-Gaussian posterior with a set of weighted samples and handles missing data in the same way — by skipping the reweighting step.
How do I assess whether missingness is ignorable?
The standard Kalman EM framework assumes observations are missing at random (MAR) — the probability of a gap does not depend on the unobserved value itself. Check whether gaps are clustered at extreme events or driven by system failures; if so, a model that explicitly accounts for the missingness mechanism is needed.
How many EM iterations are typically needed?
It depends on the problem, but convergence is usually declared when the relative change in the observed-data log-likelihood falls below a small threshold (e.g. 1e-6). Fifty to a few hundred iterations is common; initialise from multiple starting points to guard against local maxima.
Sources
- Shumway, R. H. & Stoffer, D. S. (2000). Time Series Analysis and Its Applications. Springer. ISBN: 978-0387989501
- Harvey, A. C. (1989). Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge University Press. ISBN: 978-0521405737
How to cite this page
ScholarGate. (2026, June 3). Kalman Filter for State-Space Models with Missing Observations. ScholarGate. https://scholargate.app/en/bayesian/kalman-filter-with-missing-data
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Bayesian Inference with Missing DataBayesian↔ compare
- EM AlgorithmStatistics↔ compare
- Kalman FilterBayesian↔ compare
- Particle Filter with Missing DataBayesian↔ compare
- Sequential Monte CarloBayesian↔ compare
- State Space ModelEconometrics↔ compare