Model Calibration
Also known as: Classifier Calibration, Probability Calibration, Score Calibration, Model Kalibrasyonu
Model calibration is a post-hoc technique that adjusts the probability outputs of a trained classifier so that predicted confidence scores match empirical outcome frequencies. A classifier is said to be perfectly calibrated if, among all predictions made with confidence p, exactly a fraction p of them are correct. Systematic miscalibration of modern deep neural networks was rigorously documented by Guo et al. (2017), who showed that networks trained with standard cross-entropy loss tend to be overconfident, and proposed temperature scaling as a simple, effective remedy.
Key highlights
- Temperature scaling adds only one parameter and cannot reduce classification accuracy, making it extremely lightweight.
- ECE provides an interpretable, actionable diagnostic that quantifies miscalibration before and after correction.
- Post-hoc calibration is model-agnostic and can be applied to any pre-trained classifier without retraining.
- Well-calibrated probabilities enable reliable threshold selection and expected-value calculations in downstream pipelines.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use model calibration whenever predicted probabilities will feed downstream decisions that depend on their face value, such as medical risk stratification, fraud scoring thresholds, or cost-sensitive classifiers. It is particularly important for deep neural networks, which Guo et al. (2017) demonstrated become increasingly overconfident as model depth and width grow. Calibration assumes a representative held-out set; it cannot correct for distribution shift between the calibration and deployment distributions. When prediction intervals rather than point probabilities are needed, consider conformal prediction instead.
Strengths & limitations
- Temperature scaling adds only one parameter and cannot reduce classification accuracy, making it extremely lightweight.
- ECE provides an interpretable, actionable diagnostic that quantifies miscalibration before and after correction.
- Post-hoc calibration is model-agnostic and can be applied to any pre-trained classifier without retraining.
- Well-calibrated probabilities enable reliable threshold selection and expected-value calculations in downstream pipelines.
- Calibration requires a held-out validation set; small calibration sets produce unreliable temperature estimates.
- Temperature scaling assumes a single global scalar suffices, which can fail when miscalibration is class- or region-specific.
- ECE is sensitive to the number and width of bins, making cross-study comparisons noisy without standardization.
- Calibration does not improve sharpness or discriminative ability; a poorly discriminating model remains so after calibration.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
Does calibration affect classification accuracy?
Temperature scaling does not change the rank order of class probabilities, so top-1 accuracy is unchanged. Other calibration methods such as isotonic regression may in rare cases alter the argmax prediction, but their primary goal is probability accuracy rather than classification accuracy, and any accuracy impact is typically negligible.
How do I know if my model needs calibration?
Plot a reliability diagram: group predictions by confidence bin and plot mean confidence against empirical accuracy. A perfectly calibrated model follows the diagonal. A curve consistently above the diagonal signals overconfidence; one below signals underconfidence. Compute ECE as a single summary statistic; values above roughly 0.05 are commonly considered non-trivial miscalibration in practice.
When should I use isotonic regression instead of temperature scaling?
Isotonic regression is a non-parametric monotone function and can capture complex, non-uniform miscalibration across the probability range. Prefer it when the reliability diagram shows a non-monotone or irregular pattern. However, isotonic regression requires more data to fit reliably, and temperature scaling is usually preferred when the calibration set is small or when simplicity and interpretability are priorities.
Sources
- 1.Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. International Conference on Machine Learning, 1321–1330.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 2). Model Calibration. ScholarGate. https://scholargate.app/machine-learning/model-calibration