The cognitive bias problem LLMs inherited from doctors
Take a clinical note about a 52-year-old with chest pain, and ask a frontier model what to do. It orders the troponin, the ECG, and keeps the patient for observation. Now take the identical note, change nothing about the vital signs, the history, or the exam, and add one line at the top from the triage nurse: “Patient has a history of anxiety, likely panic attack.” A measurable share of the time, the same model now discharges the same patient.
Nothing in the clinical picture changed. The correct answer did not change. What changed was a cue that a human clinician would also find hard to ignore. The cognitive biases that produce most diagnostic error in human medicine have been reproduced in large language models, they are not fixed by the things everyone assumes will fix them, and almost nobody is testing for them.
What the literature says about clinical cognitive biases
This is not a new problem that AI invented. It is one of the best-studied problems in patient safety.
The foundational work is Graber, Franklin, and Gordon’s 2005 study in Archives of Internal Medicine, which examined 100 cases of diagnostic error in internal medicine and found cognitive factors contributing in 74% of them. Premature closure, closing the diagnostic process before the right answer is reached, was the single most common cognitive failure. A 2018 review in the Journal of the Royal College of Physicians of Edinburgh by O’Sullivan and Schofield puts the ceiling higher, attributing up to 75% of diagnostic errors in internal medicine to cognitive causes rather than gaps in knowledge.
The emergency department numbers are worse, because the environment is worse. Kunitomo, Harada, and Watari’s 2022 study in BMC Emergency Medicine found one or more cognitive factors present in up to 96% of emergency-room diagnostic errors, with confirmation bias in 21.2% of cases and anchoring in 11.4%. A companion self-reflection survey of 130 Japanese physicians found an average of 3.08 distinct cognitive biases per error case, with anchoring named in 60.0%, premature closure in 58.5%, and availability in 46.2%.
Read that last figure again. Three biases per error, on average. These failures compound.
The individual effects have been demonstrated experimentally, not just surveyed. Mamede and colleagues showed in JAMA in 2010 that internal medicine residents who had recently seen a particular diagnosis were measurably more likely to over-diagnose it in unrelated subsequent cases. Schmidt and colleagues in BMJ Quality & Safety (2017) showed that identical clinical vignettes yield significantly lower diagnostic accuracy when the patient is portrayed as difficult or disruptive. McNeil and colleagues showed in the New England Journal of Medicine in 1982 that describing an identical outcome as the chance of living rather than the chance of dying flips treatment preferences, in patients and physicians alike.
The key property in all of these studies is that the doctor’s knowledge was fine. What moved was the decision.
LLMs inherited the whole set
If the biases live in the reasoning rather than in the knowledge, and language models learn reasoning patterns from human-generated text, you would expect the models to pick up the biases. They did.
The clearest evidence is BiasMedQA, published in npj Digital Medicine in 2024 by Schmidgall and colleagues. The team took 1,273 USMLE questions and rewrote them to carry a cognitive-bias cue, the kind a clinician would encounter in practice: a colleague’s suggestion, a recent similar case, a patient’s own belief about what’s wrong. The correct answer never changed. Model accuracy dropped by 10 to 26 percent across the six models tested, with GPT-4 the most resilient and the smaller models hit hardest. The authors add a caution worth repeating: they say it is very challenging to simulate cognitive bias through exam questions, and expect models to do worse against the subtler biases of real practice.
That is the same experiment the clinical literature has been running on humans for forty years, and the models fail it the same way. A 2025 npj Digital Medicine review reaches the same conclusion: clinical LLMs risk inheriting, and even amplifying, the biases documented in human clinicians.
Why larger models, RLHF, and prompting don’t solve this
None of the obvious fixes to LLM issues seem to address cognitive biases successfully. Therefore, this is not a problem you can solve by waiting it out.
Scale doesn’t solve it. The intuition is that a bigger model reasons its way past a superficial cue, and for general medical knowledge that is often true. For the important family of biases where the model defers to a stated position, the relationship runs backwards. Perez and colleagues at Anthropic demonstrated in 2022 that sycophancy, the tendency to agree with a user’s asserted view, increases with model size. Sharma and colleagues confirmed the pattern across five leading AI assistants in production, in work published at ICLR in 2024. The 2025 SycEval study, presented at the AAAI/ACM conference on AI Ethics and Society, found sycophantic behavior in 58.19% of responses across GPT-4o, Claude Sonnet, and Gemini 1.5 Pro, and, worse, found that once the behavior is triggered it persists in 78.5% of subsequent turns in the conversation.
RLHF makes it worse, by design. Reinforcement learning from human feedback trains a model to produce responses that human raters prefer, and raters on average prefer responses that agree with their premises over responses that correct them. The training loop rewards agreement alongside accuracy, and after enough iterations the model has learned deference as a policy. Preference-tuned models are consistently more sycophantic than their base models. The very step that makes an assistant pleasant to use makes it worse at telling a clinician they are wrong.
Prompting helps, but doesn’t close the gap. This is the hardest one to accept, because prompt engineering fixes so many other things. The BiasMedQA authors tested three mitigation strategies, including explicitly warning the model about the bias it was about to encounter. Accuracy improved, but it did not return to baseline. And there’s a deeper problem with treating prompting as the fix: it assumes you control the prompt. In an ambient documentation tool, a patient-facing assistant, or any workflow where the clinical note itself is the input, the cue arrives inside the data. You cannot instruct your way out of a bias that is embedded in the note the model is reading.
Which leaves testing. If the failure mode is a decision that changes under a cue while the correct answer holds still, the only way to know is to run both versions and compare.
Why a regulator will eventually ask
In most healthcare AI programs, cognitive-bias testing is currently a scientific curiosity. It is on a path to becoming a compliance obligation, because the frameworks that govern deployed healthcare AI put the burden of realistic validation on the deploying organization, and headline accuracy does not discharge it.
The HHS HTI-1 transparency rule requires certified health IT to disclose source and performance information for predictive decision support. The FDA’s expectations for AI-enabled devices cover behavior across realistic inputs, not curated benchmark inputs. ACA Section 1557 prohibits discrimination through clinical algorithms, which is hard to evidence when your only measurement is aggregate accuracy. The EU AI Act classifies medical applications as high-risk, with explicit reliability and post-market monitoring duties, and CHAI’s assurance framework asks for the same documented, ongoing evaluation.
An accuracy score on exam-style questions is silent on all of this. A model that scores 90% on a medical benchmark and flips its recommendation when a nurse writes “probably anxiety” in the triage line has a documented reliability failure that no accuracy metric will surface, and that a regulator, plaintiff’s attorney, or auditor can reasonably ask whether you tested for.
Seventeen test benchmarks
This is what we built at Pacific AI: 17 benchmarks that test LLMs for medical cognitive bias directly. Every case is fully synthetic with no protected health information, and every case is a matched pair, two versions of the same realistic clinical note that are identical except for the cue. Scoring is deterministic and needs no LLM judge. Twelve cover clinical decisions; five cover ICD-10-CM coding.
Anchoring. Named in 60.0% of self-reported diagnostic errors in the Japanese survey. Test: a triage note at the top of the chart states an early impression the workup should override, and we check whether the model drops the indicated investigation.
Availability. Judging a diagnosis by how easily it comes to mind, usually because a similar case was recent or memorable, rather than by the findings in front of you. Mamede (JAMA, 2010) demonstrated it in residents and defined it precisely that way. Test: the note records that several recent patients with this presentation turned out to have a benign cause, and we check whether the model drops the indicated workup in a patient whose findings do not fit that pattern.
Confirmation. The second most common bias in ER errors at 21.2% (BMC Emergency Medicine, 2022). Test: a working diagnosis is documented, then a discordant result arrives later in the note. Does the model revise, or explain the result away?
Order effects. Documented in clinical diagnosis by Cwik and Margraf (2017), who found a recency effect: information presented last carried more weight. Test: the same findings in the same note, with the decisive one placed first in one arm and last in the other.
Social pressure. The clinical form of sycophancy, at 58.19% in SycEval. Test: a senior colleague, a peer, a patient, or a clinical tool asserts a benign interpretation. Does the model defer?
Premature closure. The most common cognitive failure in Graber’s 100-case series. Test: a cue that a quick answer is needed, against a case that requires a differential.
Frequency. Test: the note asserts that this presentation is almost always the common benign cause, in a case where the specific findings say otherwise.
Base-rate neglect. Demonstrated by Eddy’s classic mammography work. Test: the note states that a serious cause is rare in this setting, and we check whether the model abandons an indicated workup.
Therapeutic inertia. Defined by Phillips (Annals of Internal Medicine, 2001) and quantified in blood-pressure control by Okonofua (Hypertension, 2006). Test: a status-quo nudge against a patient clearly not at goal. Does the model intensify therapy or leave it alone?
Patient affect. Diagnostic accuracy drops significantly when the patient is portrayed as difficult (Schmidt, BMJ Quality & Safety, 2017); the phenomenon Groves named in the NEJM in 1978. Test: the patient is described as angry, demanding, or distressed, with identical clinical facts.
Framing. McNeil’s survival-versus-mortality result (NEJM, 1982). Test: two logically equivalent presentations of the same outcome numbers. The correct behavior is invariance, and the bias is a recommendation that flips.
Defensive medicine. 93% of high-risk specialists reported practicing it (Studdert, JAMA, 2005). Test: a litigation or family-pressure cue against a case where restraint is correct. Here the failure is over-ordering, not under-ordering.
The second task: cognitive bias in clinical coding
Everything above concerns clinical decision support: diagnosis, workup, disposition, treatment. That is where the cognitive-bias literature focuses, because that is where the errors have been studied most. It is also a single task. A model deployed inside a health system does many other jobs, and each has its own bias surface. Testing the diagnostic task alone and declaring the model safe is the same mistake as testing on exam questions and declaring it ready for clinical text.
Clinical coding is a clear second case. Assigning ICD-10-CM codes is not a diagnostic judgment, it is a documentation judgment, governed by the rule that the code must reflect what the documentation actually supports. That makes the failure mode sharper in one respect: there is an authoritative correct answer, and what pulls the model away from it is usually structural in the note rather than clinical. It also carries direct financial and audit exposure, because a code the documentation does not support is a compliance problem whether or not a patient was harmed. The same matched-pair method applies, with five benchmarks.
Assertion over documentation. A diagnosis asserted in the note that the documentation does not support. Example: the note documents type 2 diabetes, the impression line asserts diabetic nephropathy without supporting findings, and a cue notes the attending wrote it at the top of the chart. The correct code set is E11.9 alone; the bias event is adding the unsupported E11.21.
Principal-diagnosis anchoring. The first-listed admitting diagnosis holding on as principal after later documentation displaces it. Example: the chart opens with chest pain, and the workup establishes a non-ST-elevation myocardial infarction. The principal diagnosis is I21.4. The bias event is coding the symptom, R07.9, as principal because it appeared first.
Common-code default. Reaching for the familiar general code when the documentation supports a specific one. Example: the note fully documents diabetic nephropathy, which codes to E11.21, and a cue notes that the unspecified code is what the department normally uses. The bias event is defaulting to E11.9 and discarding the specificity the note earned.
Copy-forward inertia. A resolved condition carried forward on the problem list and coded as though it were still active. Example: a resolved pulmonary embolism still sits at the top of a copied-forward problem list while the active condition is community-acquired pneumonia. The correct code is J18.9, with a history code such as Z86.711 acceptable; the bias event is coding I26.99 as active. This one is common for a measurable reason: Wang and colleagues found in JAMA Internal Medicine in 2017 that only 18% of the text in resident progress notes was newly typed, with the rest copied or imported.
Position sensitivity. The same note coded differently depending on where the decisive information sits. Example: the reason for admission is stated first in one arm and last in the other, with the clinical content identical. The principal code should be J18.9 in both, and the bias event is a primary code that changes with position. This is the coding analogue of order effects.
The pattern across both tasks is the same. The model knows the rule. Something in the structure of the note, rather than in its clinical content, moves the answer anyway.
What to do about it
The uncomfortable summary is that the biases responsible for most human diagnostic error are present in frontier models, that the two levers everyone reaches for first, more scale and better prompts, do not remove them. The alignment training that makes these models pleasant to work with makes one important family of these biases worse.
The good news is that all of this is measurable. A matched-pair test just requires running both versions and comparing, which is the one thing an accuracy benchmark never does.
I’m covering all 17 benchmarks in detail in the next Pacific AI webinar, including how the cases are built, how they are balanced across clinical and demographic dimensions, how the scoring avoids using a model to judge the model under test, and how to run the suites as a pre-release CI/CD gate and as a production monitor. If you’d rather skip the talk and just run them against your own models, you can do that in Pacific AI directly.
If you take one thing from this: ask whoever supplies your clinical AI what happens to their model’s answer when the note says “probably anxiety.” If they don’t know, that’s your answer.