What an Explanation in Medical AI Must Prove

Key points#

• Define who needs an explanation and what decision it should support before selecting an explanation method.
• Separate system transparency, global model behavior, and reasons offered for one output.
• Test fidelity and stability. A plausible visualization or paragraph is not validation.
• Evaluate the human and model together, including appropriate reliance when the model is right and when it is wrong.
• Communicate intended use, performance, uncertainty, known gaps, and failure conditions even when model logic cannot be reduced to a simple story.

Begin with the question the explanation must answer#

“Explain the model” sounds like one request, but different people need different information: a clinician reviewing one recommendation may ask which patient data affected it and whether a small correction would change it. A patient may need to understand the purpose, consequence, options, and role of human judgment. An engineer may need a global account of feature use and failure patterns. A governance group may need training-data provenance, subgroup performance, change history, and audit records.

One display cannot satisfy all of those needs. An explanation should therefore be specified like any other component:

• intended audience;
• decision or task;
• timing in the workflow;
• information shown and withheld;
• expected user action;
• known limits;
• method used to validate understanding and consequences.

The FDA, Health Canada, and UK regulator principles place explainability inside a broader idea of transparency. Intended use, target population, inputs, outputs, performance, limitations, workflow, updates, and monitoring can matter more to safe use than an attractive account of internal logic.

Separate three layers of understanding#

System transparency#

This layer describes what the tool is for and how it is governed, and it includes the clinical purpose, intended users, evaluated population, data sources, outcome definition, performance measures, uncertainty, contraindicated uses, model version, update process, and route for reporting problems. A system can be transparent at this level even when its internal model is mathematically complex.

Global model behavior#

Global interpretation asks what patterns the model generally learned. Examples include coefficients, decision rules, partial dependence, aggregate feature effects, prototypes, or summaries of behavior across carefully designed test cases; these methods can reveal broad reliance on an unexpected variable or poor behavior outside a common range. They do not necessarily explain one person’s output.

Local explanation#

A local method addresses a particular prediction. It may assign feature contributions, highlight image regions, present similar examples, give a counterfactual change, or generate a text rationale. Local approximations can be useful diagnostic tools, but they are not automatically faithful descriptions of a complex model.

Confusing these layers produces overclaiming. A model card is not an explanation for one result. A saliency map does not establish how the system behaves across a population. A natural-language rationale may help you inspect a recommendation without revealing the computation that generated it.

Know what each explanation method can and cannot claim#

An inherently interpretable model exposes a structure such as a sparse equation, small decision tree, rule list, or constrained additive function, and its logic may be more directly inspectable, although data quality, interactions, transformations, and implementation still require scrutiny. Simplicity does not guarantee validity.

Post hoc methods operate after a model produces an output. Common forms include:

• feature attribution, which assigns contributions to input variables under a selected baseline and mathematical rule;
• saliency, which marks regions of an image associated with an output;
• example-based explanation, which presents training or reference cases judged similar;
• counterfactual explanation, which identifies a change associated with another output;
• surrogate models, which approximate complex behavior globally or near one case;
• generated rationales, which produce a readable account related to the output and context.

Each answers a narrower question than “why did the model decide this?” Attribution depends on correlations and the chosen reference, while a counterfactual may propose a mathematically effective change that is impossible or clinically meaningless. A surrogate can approximate behavior in one neighborhood while differing elsewhere. Similar examples may reproduce biases in the reference set. Generated text can be coherent without being causally connected to the underlying computation. Do not let a method's name carry more authority with you than its validation evidence does.

Put explanations through technical tests#

NIST’s voluntary AI RMF playbook recommends testing explanation properties such as fidelity, consistency, resilience, and interpretability. A practical evaluation can ask:

Fidelity#

Does the explanation accurately represent the model or an explicitly defined approximation? Compare the explanation with controlled changes to the model and input. Report the domain over which any surrogate is accurate.

Sensitivity and invariance#

When a genuinely influential input changes, does the explanation respond? When an irrelevant change is introduced, does it remain appropriately stable? Both excessive volatility and artificial smoothness can mislead you.

Sanity under model changes#

Adebayo and colleagues tested saliency methods after randomizing model parameters or labels. Some visualizations remained similar even when the learned model had been disrupted, so the finding demonstrates why a plausible-looking heat map cannot on its own tell you that it reflects learned behavior.

Reproducibility#

Repeated runs, small numerical changes, or equivalent representations should not produce materially different accounts unless the method is explicitly stochastic and uncertainty is communicated. Version, software, parameters, baseline, and reference data should be recorded.

Actionability and feasibility#

Counterfactuals and feature changes should respect immutable attributes, biological constraints, time order, and the actual decision process. An impossible suggestion may be mathematically correct but useless.

Uncertainty#

An explanation should not appear more certain than the prediction or evidence permits. Show where data are sparse, the case is outside evaluated conditions, competing explanations fit, or the explanation method itself is unstable.

Test the human and AI as one decision system#

An explanation can increase confidence without improving judgment. Evaluation should include cases in which the model is correct, incorrect, uncertain, and used outside familiar patterns. The goal is appropriate reliance: accepting useful support, detecting error, and preserving independent review when needed.

Useful study outcomes include decision accuracy, error severity, time, information-seeking behavior, override quality, comprehension, recall of limitations, and the ability to identify out-of-scope use. Compare the explanation with a meaningful alternative, including clear performance and limitation information without a local explanation.

A 2026 randomized vignette study assigned 101 US radiologists to no large-language-model support or one of three output formats across 2,020 assessments. In that controlled setting, a detailed reasoning format was associated with a 12.2 percentage-point improvement in diagnostic accuracy relative to no model support. The same study described important limits: one specialty, selected image challenges, a single model family, no patient outcomes, and a need for field evaluation.

That result supports testing explanation formats with intended users. It does not establish that generated reasoning is a faithful transcript of model computation, nor that the format will have the same effect in routine care. Human usefulness and mechanistic fidelity are separate claims, and both can matter.

Watch for explanation-specific hazards#

Explanations can disclose sensitive training examples or invite reconstruction of protected data. They can also reveal how to manipulate a model. Example-based methods need privacy review; interactive tools need security testing; audit logs need access controls and retention rules.

Interface design can create anchoring. A vivid heat map or long rationale may dominate more reliable information because it is easier to process. Show essential limitations at the moment of use, not behind a distant link. Make model uncertainty and out-of-scope conditions at least as visible as the explanation.

Group effects deserve evaluation too. Familiarity with visualizations, language proficiency, accessibility, and clinical background can alter how an explanation is understood. Average comprehension does not establish usability for every intended audience.

A disciplined explanation review#

Before you accept an explanation claim, ask:

1. Who is the audience and what must they decide?
2. Is this system transparency, global interpretation, or a local account?
3. Which mathematical or generative method produced it?
4. What evidence supports fidelity, stability, and feasibility?
5. How does uncertainty appear?
6. Was it tested with intended users on correct and incorrect outputs?
7. Did it improve appropriate reliance rather than confidence alone?
8. Are privacy, security, subgroup, accessibility, and version risks controlled?

The standard is not whether an explanation looks convincing. It is whether a defined audience can use validated information to make a better, appropriately cautious decision. The articles on clinical prediction model appraisal and AI in radiology provide related tests for performance and use context.