NRI and IDI Reclassification Metrics, and Their Pitfalls

Net reclassification improvement, or NRI, and integrated discrimination improvement, or IDI, were created to answer a reasonable question: when a new marker is added to a risk model, do predictions move in a more useful direction? The difficulty is that a favorable statistic does not by itself show better calibration, better decisions, or better patient outcomes.

The safest reading is to treat NRI and IDI as supplementary descriptions. A model still needs external validation, calibration assessment, discrimination, overall accuracy, and decision-focused evaluation.

Key points#

• Category-based NRI depends on risk thresholds that should correspond to real decisions.
• Continuous NRI counts any movement in the preferred direction, however small, and can produce large values without meaningful improvement.
• IDI compares average separation of predicted risks between people with and without the outcome.
• Both metrics can be distorted by overfitting, miscalibration, censoring, and evaluation on development data.
• Reclassification is persuasive only when the underlying models are valid and the changed category would alter care beneficially.

Begin with two models and one target outcome#

Suppose you have a baseline model that predicts a person's five-year probability of an outcome from age, blood pressure, and smoking. An expanded model adds a biomarker. Both models output probabilities. Reclassification asks how those probabilities change.

For people who later have the outcome, upward movement seems directionally correct. For people who remain outcome-free, downward movement seems correct. That intuition powers NRI. IDI instead compares how much the average predicted risk differs between event and nonevent groups under each model.

Before calculating either, the outcome horizon, target population, prediction time, and handling of competing events must match, and a five-year model cannot be judged from a two-year follow-up without methods that correctly handle censoring. A model for first events should not silently mix recurrent events; the baseline model matters too: adding a marker to a weak or obsolete comparator can create apparent improvement that disappears against a well-specified standard model.

Category-based NRI counts crossings that matter only if categories matter#

Category-based NRI starts with risk bands, such as below 5%, 5% to below 10%, and at least 10%. It then counts movement across a boundary.

For participants with events, the event component is:

proportion moving up minus proportion moving down

For participants without events, the nonevent component is:

proportion moving down minus proportion moving up

Their sum is the NRI. Reporting the two components separately is essential. A positive total could be driven by improved classification of events while classification of nonevents worsens, or the reverse. Those consequences may have different clinical value.

Risk boundaries must be chosen without reference to which cutoffs make the statistic favorable. Ideally each boundary maps to an action supported by evidence, such as offering a preventive treatment or ordering a confirmatory test. If the same care follows on both sides, crossing the boundary has no immediate decision value.

Results can change sharply when categories change: a prediction moving from 9.9% to 10.1% counts, while one moving from 5.1% to 9.9% does not, even though the second probability change is much larger. Show the complete reclassification table so you can see how many people crossed each boundary and by how much.

Continuous NRI removes categories and creates a new problem#

Continuous, or category-free, NRI counts any increase as correct for an event and any decrease as correct for a nonevent, so a move from 10.000% to 10.001% counts the same directionally as a move from 10% to 30%.

Because almost every prediction can move slightly when a variable is added, continuous NRI can become numerically large even when the new model changes little that matters. Under common conditions it has a positive expectation when a predictor is associated with outcome, but the magnitude does not translate naturally into a percentage of people correctly reclassified.

The common phrase “the model reclassified 40% of patients correctly” is often an incorrect interpretation of a continuous NRI of 0.40, because the statistic is a sum of two net proportions, not the fraction of all individuals whose clinical category became correct. Continuous NRI is especially vulnerable when models are miscalibrated or compared on the data used to develop them. It should never substitute for probability-level calibration plots and decision analysis.

IDI measures average separation#

For a binary outcome, the discrimination slope is the mean predicted risk among people with the outcome minus the mean among people without it. IDI is the expanded model's discrimination slope minus the baseline model's slope.

If the new model raises average predictions for events and lowers them for nonevents, IDI is positive. Unlike the area under the receiver operating characteristic curve, IDI is sensitive to changes in the actual predicted probabilities, not only ranking.

That sensitivity can be useful, but IDI has no universally meaningful scale. An IDI of 0.01 means the average event-nonevent separation increased by one percentage point. Whether that is important depends on outcome frequency, horizon, calibration, decisions, and the distribution of individual changes.

IDI also does not tell you where the change occurred. A small average can hide large changes in a few people, and a larger average may occur far from any decision threshold. Displaying prediction distributions or paired changes adds needed context.

Calibration is a prerequisite#

Calibration asks whether predicted probabilities agree with observed frequencies, and a model that assigns 20% risk to a group should see roughly 20% experience the outcome over the stated horizon, within uncertainty.

NRI can reward movement in a direction even if probabilities become less accurate. Imagine an expanded model that raises risk for almost every eventual event but raises it far too much. Event NRI may improve while the model systematically exaggerates risk. The same problem can affect IDI.

At minimum, compare calibration-in-the-large, calibration slope, and a flexible calibration curve in the validation data. Assess calibration overall and in clinically important groups. A single goodness-of-fit p value is not enough; it depends on sample size and does not show the direction or magnitude of error. Recalibration may improve probabilities without changing ranking. If models are recalibrated differently before reclassification analysis, that choice should be explicit and justified.

Development data create optimism#

Adding predictors almost always improves apparent fit in the same sample, even when some improvement captures noise. Reclassification metrics calculated there inherit that optimism. Splitting a small dataset once into training and test sets can also produce unstable results.

Use bootstrap or cross-validation for internal validation during development, then evaluate the locked model in external data from another place, time, or both. All preprocessing, variable selection, missing-data handling, and thresholds must be carried into validation without learning from outcomes there.

For survival outcomes, use methods that account for censoring and competing risks, because excluding people without complete follow-up can create selection bias. The paper should make clear the estimator and the assumptions, not merely label the result a survival NRI. Confidence intervals and uncertainty are required. Reclassification tables may contain small cells, and bootstrap procedures need to repeat the full model-building process if development uncertainty is part of the target.

Limits: clinical utility is a separate test#

A model is useful when it improves decisions, not simply statistics. Decision-curve analysis estimates net benefit across risk thresholds by weighting false positives against true positives. It compares model-guided strategies with alternatives such as treating everyone or no one.

Even net benefit is not a patient-outcome trial. It relies on a threshold representing the tradeoff between missed outcomes and unnecessary intervention. The treatment triggered by the model must itself be beneficial for the target group, and costs or burdens may need explicit analysis. A useful reclassification report states how many people cross each action threshold, their observed outcome rate, which action changes, and the expected benefits and harms. Without that bridge, “improved reclassification” remains a statistical claim.

A reader's appraisal sequence#

Ask yourself these questions in order:

1. Was the baseline model appropriate and applied correctly?
2. Were both models evaluated on external data with adequate outcomes and follow-up?
3. Are predicted probabilities well calibrated overall and by key groups?
4. Were risk categories prespecified and tied to real actions?
5. Are event and nonevent NRI components and the full table reported?
6. If continuous NRI is used, are magnitudes of individual probability changes shown?
7. Is IDI translated as a change in average separation rather than a percent correctly classified?
8. Do discrimination, overall error, and decision analysis tell a coherent story?
9. Were missing data, censoring, and competing events handled appropriately?
10. Would using the expanded model improve a decision enough to justify its burden?

TRIPOD+AI calls for transparent reporting of model development and evaluation regardless of whether the model uses regression or machine learning. That transparency matters more than any single performance statistic.