When a Health Algorithm Should Say I Don't Know

A health algorithm can be highly accurate on average and still be unsafe when it treats every input as answerable. The patient may differ from the development population. A laboratory unit may be wrong. An image may be cropped. A key symptom may be missing. Two data sources may conflict. The model may be close to a decision boundary where an error has serious consequences.

In these cases, the right output may be no prediction, a wider set of possibilities, a request for better information, or an urgent route to human review, and this is called abstention, rejection, selective prediction, or deferral, depending on the system.

Abstention is not a text phrase added at the end of development. It is a designed clinical and operational pathway with measurable benefits, harms, workload, and equity effects.

The forced-answer problem#

Many machine-learning systems are trained to choose among available labels, so a classifier may assign probabilities that sum to one even when the image is corrupted or the true diagnosis is absent from the label set. A language model may generate fluent text for a contradictory prompt because generation is its default behavior.

The highest score can therefore mean “least unlikely among these choices,” not “supported well enough for care.” A probability of 0.72 may look precise while being badly calibrated in the hospital where you deploy it. Forcing an answer transfers hidden uncertainty to the user. Because polished output can feel authoritative, the person may act before noticing that the input was incomplete or outside scope.

Abstention is a decision, not silence#

In selective prediction, the system produces an answer only when a selection rule is satisfied. Otherwise it defers. Coverage is the proportion of cases answered. Selective risk is the error rate or expected loss among those answered.

Lowering coverage can reduce error among accepted cases, but only if the uncertainty signal ranks hard cases usefully. A trivial system could claim perfect accuracy by answering one obvious case. A useful system must achieve an acceptable balance between reliable answers and workable deferral volume, and the abstention output should state what failed, when possible: inadequate image quality, missing required variables, unsupported population, conflicting data, out-of-range value, or uncertainty above a validated threshold. It should also state the next action.

Situations that should trigger deferral#

The first category is invalid input. Examples include the wrong imaging view, motion artifact, a specimen-quality failure, impossible dates, mismatched units, missing laterality, or absent required fields. Quality checks should occur before prediction.

The second is an out-of-distribution case. A model trained on adult chest radiographs should not silently analyze a neonatal study. A dermatology system may not be validated for mucosal lesions. A sepsis model trained before a major laboratory-platform change may face altered data.

The third is unresolved ambiguity. Two likely classes may have similar scores, a measured value may lie close to a consequential threshold, or the record may support incompatible interpretations. High-stakes near-boundary cases often merit another test or review even if the model can rank them.

The fourth is a safety override. Certain symptoms or findings can require urgent escalation regardless of model confidence, such as signs of stroke, severe hypoxemia, or an immediately life-threatening laboratory value; a decision rule can route these cases without treating the model as the final authority.

Confidence is not the same as correctness#

Neural networks can be overconfident. Softmax scores and token probabilities are properties of a fitted model, not direct measurements of clinical truth. Calibration asks whether events predicted at 80 percent occur about 80 percent of the time in an appropriate set of cases.

A model can be calibrated overall and miscalibrated in a subgroup. It can be calibrated at launch and drift later. Prevalence changes alone can alter predictive values, while a new scanner, referral route, or documentation template can change input distributions.

Uncertainty methods such as ensembles, conformal prediction, Bayesian approximations, distance measures, or auxiliary quality models can help. None is universally sufficient. The choice and the threshold need validation under the shifts you expect your system to face.

Dataset shift is the real test#

Internal test data often resemble training data. Real deployment introduces another institution, season, patient mix, device, workflow, and pattern of missingness. It may also change behavior because clinicians know your model is present.

Evaluate the abstention rule on external sites and on stress sets you build deliberately. Tests can include corrupted inputs, rare conditions, new devices, changed disease prevalence, missing variables, and populations underrepresented in development.

The NIST AI Risk Management Framework organizes risk work around govern, map, measure, and manage. It is voluntary guidance, not a medical-device clearance or a universal legal mandate. Its emphasis on context, validity, reliability, safety, transparency, and monitoring supports a lifecycle approach to uncertainty.

Intended use sets the boundary#

Your intended use should identify the task, the users, the patient population, the input requirements, the clinical setting, the role in the workflow, and the decisions it informs. “Supports diagnosis” is too broad. A chest-image triage tool and a definitive diagnostic system have different error costs and fallback needs.

The final 2025 IMDRF Good Machine Learning Practice principles call for a clearly defined intended use, representative data, attention to the human-AI team, suitable testing, and monitoring across the product lifecycle. These are guiding principles, not a claim that one abstention formula satisfies regulators. When a case violates intended-use conditions, the system should refuse the unsupported task or display an unmistakable limitation. Allowing the user to click past a warning without consequence turns scope into decoration.

Regulatory guidance is lifecycle-oriented#

FDA's January 2025 draft guidance for AI-enabled device software functions proposes recommendations for marketing submissions and total-product-lifecycle risk management. It is explicitly draft, not for implementation, and contains nonbinding recommendations.

It asks a sponsor to describe the inputs, the outputs, the intended users, the use environment, model development, performance, risk controls, and monitoring, so an abstention function that affects output availability, user action, or safety belongs in your system description and your validation. FDA's joint transparency principles emphasize relevant information for users and performance of the human-AI team. A deferral message should be understandable at the moment of decision, not buried in a technical manual.

Designing the fallback before the threshold#

“Refer to a clinician” is incomplete if no one receives the referral, the queue has no response time, or the reviewer lacks the original data. Your fallback must name who owns the case, what information travels with it, how urgency is assigned, and what happens when review is unavailable.

For low-quality data, the next action may be to repeat an image or specimen. For missing information, it may be a targeted question. For a high-risk uncertain diagnosis, it may be immediate specialist review. For a low-risk administrative prediction, a manual workflow may be enough.

Capacity constrains threshold choice. If 40 percent of cases defer to a team that can review 5 percent, delay becomes the new harm. Workflow simulation should estimate volume by hour, site, and patient group before launch.

Human review can also fail#

Humans can anchor on an AI suggestion, overlook a warning, assume a deferred case is probably negative, or become less vigilant after many correct outputs. Reviewers may also face time pressure and incomplete data.

If the system shows its tentative answer during deferral, that answer can bias review. If it hides all information, reviewers may lose useful signal. The interface choice should be tested rather than assumed. WHO's ethics and governance guidance calls for human autonomy, safety, transparency, accountability, inclusiveness, and responsive sustainability. Human control is meaningful only when the person has authority, information, time, and a functioning route to act.

Metrics beyond overall accuracy#

Report a risk-coverage curve, not a single accuracy number. Show sensitivity, specificity, predictive values, calibration, and clinically weighted harm at several deferral thresholds. Include the abstention rate and error rate among answered cases.

Then evaluate the full path: reviewer workload, time to resolution, repeat-test rate, override frequency, unreviewed queue, downstream treatment, and patient outcomes. Compare with usual care and with the model operating without abstention where ethical. Subgroup analysis should show both answered-case performance and deferral rates. A model can appear fair among accepted cases while sending far more patients from one group into a slow manual queue.

Language models need explicit evaluation#

Generative systems pose a distinct problem because they can produce confident prose without calibrated case-level probabilities. Self-reported confidence and phrases such as “I am not sure” are not validated uncertainty measures by default.

The 2026 MedQAbstain benchmark tested language models on medical questions under constructed uncertainty and found systematic overcommitment, including when key information was withheld. A benchmark is not a clinical trial, but it shows why ordinary question-answer accuracy does not establish safe abstention behavior. Evaluation should include missing facts, contradictory records, impossible premises, questions outside intended use, requests for individualized treatment, and adversarially plausible misinformation. The required behavior may be clarification, source retrieval, escalation, or refusal.

Explanations do not cure uncertainty#

A feature-attribution map, rationale, or cited sentence may help a user inspect output. It can also create unwarranted confidence. An explanation can be coherent even when the model used a spurious signal or the case lies outside its validated population.

Explainability and abstention solve different problems. Explanation addresses how an output can be understood or scrutinized. Abstention addresses whether the system should issue that output at all. Both require evaluation with actual users. The interface should distinguish “input invalid,” “outside scope,” “uncertainty too high,” and “urgent rule triggered.” Collapsing all four into “error” deprives users of the action they need.

Monitoring after launch#

Track input quality, missingness, score distributions, calibration, coverage, subgroup deferral, reviewer response time, overrides, and adverse events. Compare values with a prespecified baseline. Investigate sudden changes and slow drift.

Model updates, software changes, new clinical protocols, and new hardware can invalidate thresholds. Revalidation should cover the selection rule and fallback workflow, not only the predictor. A safe stop mechanism is essential. If monitoring shows unmanageable deferral, degraded performance, or unequal harm, the organization should be able to narrow use or suspend the system while maintaining care.

A practical governance checklist#

Define which cases the algorithm may answer. List required inputs and hard quality failures. Specify the uncertainty method and the clinical loss that shaped your threshold. Validate across sites, shifts, and subgroups. Publish coverage and risk together.

Name the fallback owner, service level, escalation route, and documentation. Test the interface with users. Monitor both answered and deferred cases. Record threshold changes and preserve audit logs.

Most important, treat “I don't know” as a safe transition, not an endpoint. The patient still needs an answer from the health system.

References#

1. NIST AI Risk Management Framework 1.0
2. IMDRF 2025 Good Machine Learning Practice
3. FDA draft AI-enabled device lifecycle guidance
4. FDA medical-device transparency principles
5. WHO ethics and governance of AI for health
6. MedQAbstain benchmark