How to Read a Calibration Plot

A calibration plot asks a concrete question: among people assigned a 20% risk, does the outcome occur in about 20%? The horizontal axis shows predicted probability. The vertical axis shows observed outcome frequency. A model whose probabilities agree with outcomes follows the 45-degree diagonal.

That simple description hides several judgments. The plot depends on who was evaluated, how outcomes were defined, how long people were followed, and how the curve was estimated. It also needs uncertainty. A smooth-looking line is not proof that a model will support good decisions.

Start with the diagonal#

The ideal diagonal runs from 0% predicted and 0% observed to 100% predicted and 100% observed. It is a reference, not a fitted result.

Consider a model used to estimate a five-year outcome. If 500 people receive predictions near 30% and 180 experience the outcome, the observed frequency is 36%. That point lies above the diagonal. The model underpredicted risk in that group. If only 120 experienced the outcome, the observed frequency would be 24%, below the diagonal, meaning the model overpredicted risk.

Words such as “underconfident” and “overconfident” can help, but directional language is safer. State whether predicted risk was lower or higher than observed risk. The clinical consequence then depends on the decision you are using the number for: overprediction might cause unnecessary testing at one threshold, and underprediction might deny preventive treatment at another.

The constructed bins make the direction easy to see. They should not be treated as evidence about any real model. In an actual evaluation you would need confidence intervals, and the choice of bins could change the apparent shape.

Curves usually tell more than bins#

Grouping predictions into deciles or other bands is intuitive, but bin boundaries are arbitrary. Two people with predictions of 19.9% and 20.1% can land in different groups, while people at opposite ends of a broad group are averaged together. Small groups are noisy. Large groups conceal local problems.

A flexible calibration curve uses the individual predictions to estimate how observed probability changes across the scale, and it can reveal an S-shape, a local departure near a treatment threshold, or poor performance at the extremes. The smoothing method and amount of data still matter. Sparse tails can produce an unstable curve, so the graph should show the distribution of predictions and an uncertainty band. A plot that covers only 2% to 18% predicted risk tells you nothing about calibration at 50%. Axes should reflect the observed prediction range without visually magnifying trivial departures.

Three numerical summaries add context#

Calibration-in-the-large compares average predicted risk with average observed risk. For a binary outcome under common logistic-model conventions, an ideal calibration intercept is zero. A positive intercept generally signals underprediction overall, and a negative intercept signals overprediction overall. The exact interpretation depends on how the statistic was fitted and reported.

The calibration slope evaluates spread. A slope of one is ideal. A slope below one often means predictions are too extreme, the common pattern after overfitting, with high predicted risks too high and low predicted risks too low, while a slope above one suggests the predictions are not spread far enough. Neither number tells you where the local deviations sit, so both belong beside a curve.

The Brier score is the mean squared difference between predicted probabilities and observed binary outcomes. Lower is better, but it combines calibration and discrimination and depends on outcome frequency. A single Brier score cannot replace a calibration plot.

Calibration is not discrimination#

Discrimination describes ranking. For a binary outcome, the area under the receiver operating characteristic curve asks how often a randomly selected person with the outcome receives a higher score than a person without it. It does not ask whether 30% means 30%.

A model can preserve an excellent ranking after moving to a population with a different event rate, yet systematically overpredict or underpredict absolute risk. The reverse is also possible: a model can be calibrated on average while barely separating higher-risk from lower-risk people, so a credible evaluation gives you all three, calibration, discrimination, and the consequences at clinically relevant thresholds.

Check the population, outcome, and clock#

Calibration belongs to a setting, not permanently to a model. Differences in referral patterns, eligibility, measurement, treatment, and outcome prevalence can shift it, so a hospital model evaluated in community clinics may encounter a narrower illness range, and a risk score trained before a new preventive therapy may overpredict events once care improves. The curve was drawn somewhere.

The predicted and observed outcomes must use the same definition and time horizon: a ten-year cardiovascular prediction cannot be checked against five-year follow-up without methods that handle censoring and the intended horizon. A diagnostic probability collected at presentation cannot be compared casually with any diagnosis recorded months later.

Sampling also matters. A conventional case-control sample fixes the number of cases and controls, so its raw event proportion does not represent population risk. Absolute calibration requires appropriate sampling information or weighting. Missing outcomes, loss to follow-up, and competing events can distort observed frequencies.

Overall calibration can hide subgroup failure. TRIPOD+AI recommends reporting model performance in relevant groups. Each subgroup curve needs enough outcomes, and comparisons should be prespecified where possible. A jagged line from a handful of events should not be mistaken for a stable disparity.

Apparent performance is optimistic#

A curve drawn on the same data used to build a flexible model is usually too flattering. The model has already adapted to noise in that sample. Internal validation, such as bootstrapping or cross-validation performed across the full modeling process, estimates optimism. Evaluation in data collected at a different place or time tests transport more directly.

Even a separate test split can be weak if it is small or created after extensive trial-and-error. Ask whether preprocessing, feature selection, tuning, and threshold choice were confined to training data. Data leakage can improve both discrimination and apparent calibration.

Recalibration is a repair, not a verdict#

If ranking remains useful but baseline risk has shifted, adjusting the intercept may align the average prediction. Adjusting both intercept and slope can correct average level and spread. More flexible updating may address a curved mismatch.

Any update needs new evaluation, version control, and monitoring. Correcting probabilities in one dataset does not show clinical benefit, fairness, usability, or safety; it also cannot fix an outcome label that differs from practice, missing predictors, poor data quality, or a model used outside its intended population.

A practical reading sequence#

Name the outcome and prediction horizon first. Confirm that the evaluation population matches the use you have in mind. Note sample size, outcome count, censoring, missingness, and the prediction range. Then read the diagonal, flexible curve, uncertainty, intercept, and slope. Look separately at discrimination and decision thresholds.

Finish by asking what an error would do to the person the number is about, because a 5-point mismatch around a treatment threshold may matter more than a larger mismatch where no decision changes. Calibration is necessary for probability-based decisions, but usefulness depends on the benefits and harms attached to those probabilities.

References#

1. Van Calster and colleagues, calibration in predictive analytics
2. Steyerberg and colleagues, prediction-model performance measures
3. Huang and colleagues, tutorial on calibration measurements and models
4. Collins and colleagues, TRIPOD+AI reporting statement