Model Calibration

Model calibration is a post-hoc technique that adjusts the probability outputs of a trained classifier so that predicted confidence scores match empirical outcome frequencies. A classifier is said to be perfectly calibrated if, among all predictions made with confidence p, exactly a fraction p of them are correct. Systematic miscalibration of modern deep neural networks was rigorously documented by Guo et al. (2017), who showed that networks trained with standard cross-entropy loss tend to be overconfident, and proposed temperature scaling as a simple, effective remedy.