When a Confident AI Makes a Good Clinician Worse

The most striking finding in the newest research on medical AI is not that the tools make mistakes. Everyone expected that. It is that a confident, wrong suggestion can pull an experienced clinician's reasoning down below where it would have landed with no tool at all. In a randomized trial, physicians who had already completed formal AI-literacy training scored worse on clinical reasoning when a language model fed them plausible but incorrect advice than colleagues who got sound advice from the same tool. The tendency behind that drop has a name, automation bias, and recent studies let us describe its size and its cause rather than simply warn that it exists.

Questions about your own care belong with your own clinician.

Key points#

• Automation bias is over-trust in an automated answer, either acting on a wrong one or failing to act because the system stayed silent.
• In a randomized trial, deliberately flawed AI advice cut physicians' diagnostic-reasoning scores by roughly 14 percentage points, even after about 20 hours of AI-literacy training.
• A separate trial in medical students found the same lopsided result: wrong AI guidance hurt more than correct guidance helped.
• Stated confidence, from the model or the human, did not reliably separate right answers from wrong ones.
• The fix the investigators point to is structural: workflow design, verification steps, and real human oversight, not the hope that a trained professional will catch the error.

Two different ways trust goes wrong#

It helps to separate two failure modes that often get blurred together.

The first is automation bias in the narrow sense: over-relying on a specific recommendation. That can mean acting on a wrong suggestion (a commission error) or missing something because the system said nothing (an omission error). It shows up on the hard individual case where the machine happens to be wrong.

The second is automation complacency, a slower erosion of vigilance that builds up when a tool is usually right. It shows up not on one dramatic case but as a gradual decline in independent checking across many routine ones. Both push in the same direction. The human stops fully verifying the output, and the moment they stop is the moment a rare error slips through unexamined.

What happened when trained physicians were tested#

The sharpest evidence comes from a randomized study of 44 physicians, all of whom had completed roughly 20 hours of formal AI-literacy training. Each worked through six clinical vignettes with optional help from a large language model. For half the participants the model's answers were left untouched. For the other half, three of the six answers were deliberately corrupted. That design isolates the variable that actually matters, which is not whether AI is in the room but whether the AI is wrong.

Because the participants were, by construction, a best case, the results are hard to shrug off. Overall diagnostic-reasoning scores fell from 84.9 percent in the sound-advice group to 73.3 percent in the flawed-advice group, an adjusted gap of about 14 percentage points. Accuracy on the single top diagnosis dropped further, from roughly 90.5 percent to 76.1 percent, an adjusted gap near 18 points. Knowing how these systems work did not inoculate the clinicians against a fluent, confident, wrong answer. The authors also noted an exploratory signal that heavier habitual model users showed larger drops, exactly the pattern you would predict if routine reliance dulls independent scrutiny. That subgroup finding rests on a small sample and should be read as a hypothesis, not a verdict.

Two design choices make the study informative. The planted errors were plausible, not absurd, which mirrors how modern language models really fail: subtle, well-worded mistakes rather than obvious nonsense. And the comparison group used the same tool giving correct advice, so the measured harm traces to the wrong content and the impulse to defer to it, not to the mere presence of a computer.

The same pattern in learners#

A separate randomized study of 111 medical students found the same lopsided result in people with less clinical experience. Misleading AI explanations significantly worsened diagnostic accuracy, while correct AI explanations produced no significant gain over a plain no-explanation control. The downside from wrong guidance was larger and more reliable than the upside from right guidance.

The study added a second uncomfortable finding. Confidence was poorly calibrated. Students' own stated confidence did not reliably separate their correct answers from their wrong ones, so self-assurance was useless as a signal for when to trust the machine. For a learner still building pattern recognition, an incorrect but authoritative-sounding suggestion appears to be especially costly.

Why a polished wrong answer is the hard case#

A conceptual analysis in the journal AI and Ethics offers a mechanism that fits these numbers. When a clinician offloads part of a task to an automated aid, attention shifts away from the decision itself. Situational awareness narrows, and what the authors call learned complacency sets in, so the person monitors the tool's output less and less closely over time. The worry is not only one wrong call but a gradual thinning of the vigilance that makes oversight mean anything.

Language models sharpen the problem. Older decision-support tools tended to hand back a discrete label with a numeric confidence score, which at least flagged its own uncertainty. A language model instead produces a smooth narrative that reads like expert prose while sometimes hiding a clinically important error inside it. Think of a satellite navigation system that speaks in a calm, certain voice while routing you onto a closed road. The tone earns a trust the content has not. That mismatch between fluency and accuracy is precisely what automation bias feeds on.

What these studies can and cannot tell us#

Read the exact percentages as direction and magnitude, not fixed constants. These are experiments, mostly built on written vignettes rather than live practice, with modest sample sizes. What replicates across settings is the qualitative result, and it is the part worth remembering: confident, wrong AI advice measurably degrades human decisions, and people do not reliably catch the error on their own.

The practical lesson the investigators draw is structural rather than personal. Safety depends on how the workflow is built, on verification steps, and on genuine human oversight, not on the assumption that a trained professional will simply notice when the machine is off. One more distinction is worth keeping. Evidence that a tool raises average accuracy is not evidence that it is safe when it fails. Those are separate questions, and each deserves its own testing.

Where this leaves us#

The headline is easy to misread. The danger these studies expose is not a clumsy machine that anyone can see through. It is a smooth one that sounds right while being wrong, aimed at humans who are wired to trust a confident voice. Knowing that the tool is fallible is not enough, because the reasoning slips even in people who know it. What protects a decision is a system that expects the error and checks for it, and a clinician who keeps forming a view of their own before the screen offers one.