When the Title Outruns the Study: General-Purpose vs. Healthcare-Specific AI

A Brief Communication in Nature Medicine made the rounds last month under a headline that is hard to misread: “General-purpose large language models outperform specialized clinical AI tools on medical benchmarks.” It generated a lot of commentary, a combative public response from one of the named companies, and a request to the journal for a retraction.

The study is fine. The methods are reasonable and the statistics are careful. The problem is the title. It states a general conclusion that the study did not test and does not support. The body of the paper, to the authors’ credit, is far more careful than the headline suggests.

A familiar pattern: titles that outrun the evidence

This isn’t a problem specific to AI. Nutrition research has been living with it for decades.

Consider a 2011 paper titled “Intake of added sugars is not associated with weight measures in children 6 to 18 years.” The title states a sweeping null result. The study behind it was a cross-sectional analysis of NHANES data using a single 24-hour dietary recall per child: a one-day snapshot of self-reported eating. A design like that genuinely cannot establish that sugar is “not associated” with weight: it can’t address reverse causation (heavier kids who have already started cutting back), it can’t capture habitual intake from one recalled day, and it can’t speak to causation at all. The finding may be real within its narrow frame. The title claims something the frame can’t carry.

You can find this pattern repeatedly. A widely cited 2008 meta-analysis concluded that the association between sugar-sweetened beverages and children’s BMI was “near zero”, and even flagged its own evidence of publication bias, before later and larger syntheses found a clear positive association. Reviewers have since documented that industry-funded reviews of sugar and weight were roughly five times more likely to report no association than independent ones.

Two lessons travel from nutrition to medical AI: the scope of a title should match the scope of the evidence, and it always matters who is asking the question and how.

What the paper measured: single-turn general medical knowledge

So what did this study test? Two commercial clinical tools, OpenEvidence and Wolters Kluwer’s UpToDate Expert AI, against three frontier models, GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6. The evaluation had three stages:

• 500 MedQA questions: USMLE-style, multiple-choice medical knowledge.

• 500 HealthBench items: single-turn, free-response prompts graded for alignment with clinicians (a benchmark built by OpenAI).

• 100 “real clinical queries” (RCQ): de-identified single-turn questions that physicians had typed into NYU Langone’s HIPAA-compliant GPT instance, scored blind by 12 clinicians, producing 1,800 annotations.

The frontier models won all three. On MedQA, Gemini hit 97.4% versus OpenEvidence’s 89.6% and UpToDate’s 88.4%. On HealthBench, GPT led at 88.0 with the two clinical tools around 62. On the real-query benchmark, the three frontier models formed the top tier (3.5–3.6 on a 1–4 scale) while the clinical tools (3.17–3.24) came out about even with Google’s free auto-generated Search AI Overview (3.27).

Now look at what these three stages have in common. Every one of them is the same task: answering a single-turn, general medical question, in isolation, with no patient record attached. MedQA is an exam. HealthBench is exam-adjacent. RCQ is a real but still single-turn question. The study measured one capability three times: general medical question answering. That is a legitimate thing to measure. It is not “medical benchmarks,” and OpenEvidence and UpToDate are not all “specialized clinical AI tools.” Two products, one task family.

To the authors’ credit, the body says as much:

• They flag that MedQA and HealthBench may have leaked into training data

• They note HealthBench was built by OpenAI and that GPT-5.2 may benefit from “benchmark-developer overlap”

• They call the RCQ the primary evidence and HealthBench merely supplementary

• They frame the whole thing as “a snapshot of a rapidly evolving landscape,” adding that “deeply subspecialized medical tasks may favor more sophisticated, domain-specific adaptation.”

All of that careful hedging lives under a title that hedges nothing.

One task is not “medicine”

Why does the single-task issue matter so much? Because real clinical work is not a quiz. It’s summarizing a chart, drafting a discharge instruction, extracting a tumor stage from a pathology report, reconciling a medication list, catching an error in a note, turning a clinician’s question into a database query.

This is exactly the gap that MedHELM, the Stanford-led, open-source benchmark published in Nature Medicine, was built to close. MedHELM organizes clinical AI into a clinician-validated taxonomy of 5 categories, 22 subcategories, and 121 distinct tasks, and evaluates models across roughly three dozen benchmarks. Many of these benchmarks are deliberately private or gated, drawn from clinical operations rather than exam material, precisely so the test set can’t be memorized. Its whole premise is that near-perfect exam scores tell you very little about deployment.

You can see the same philosophy in the benchmark suite we publish, which compares John Snow Labs’ medical language models against GPT-5.4, Gemini 3.1 Pro, and Claude Opus 4.6 across 13 clinical and biomedical tasks, which overlaps heavily with MedHELM. A few tasks from these suites, with an example of what each actually asks:

• MEDEC: medical error detection and correction in clinical notes. Given a note, flag whether a sentence contains an error (wrong diagnosis, wrong drug, wrong causal organism), point to the sentence, and rewrite it. On the original benchmark, the best models scored around 70% versus about 80% for physicians.

• MedCalc-Bench: clinical calculations from a patient vignette. For example: given a note, compute the patient’s Cockcroft-Gault creatinine clearance, which requires pulling the right values from the right encounter and applying the right formula. This is the hardest task in the set: direct prompting tops out around 35% on the verified version.

• EHRSQL: turning a clinician’s plain-English question into SQL over a hospital database (”How many patients were prescribed warfarin in the last month?”). Built from questions posed by 200+ real hospital staff, it also includes unanswerable questions to test whether the model knows when to abstain.

• ACI-Bench: generating a structured visit note from a doctor–patient conversation transcript. The input is the messy back-and-forth of a real clinical encounter; the output is the note.

• MedAlign: following a clinician’s instruction against a real, longitudinal patient chart, e.g. “Summarize this patient’s past symptoms, examinations, treatments, and surgeries.” Answering it means synthesizing dozens of notes across years of one real record – something no amount of memorized literature helps with. The data is gated behind a research agreement, so it can’t be crawled, and even GPT-4 got roughly a third of these instructions wrong. The difficulty persists for newer models: a 2025 preprint, TIMER, found the strongest system it tested correct on only about 41% of MedAlign cases on its stricter longitudinal-reasoning measure, with GPT-4o and Claude 3.5 Sonnet among those evaluated – though, as a preprint, that result is not yet peer-reviewed.

These tasks reward different things than a multiple-choice exam does: information extraction, faithfulness to a source document, calibrated abstention, and structured output. A model can ace USMLE-style questions and still be mediocre at all of them. This happens often.

Why general models shine on what they’ve already seen

There’s a deeper reason the single-task design flatters general-purpose models: two of the three stages use public datasets, and public datasets get memorized.

The evidence here is not subtle. The “RABBITS” study showed that simply swapping brand and generic drug names in MedQA and MedMCQA questions – a change no clinician would even notice – dropped model accuracy by 1–10%. The authors traced the fragility to test-set contamination in pretraining data. The self-assessment-for-neurosurgeons study (from the same NYU group, notably) found that adding distractions in text cut accuracy by as much as 20.4%.

The “None of the Above” analysis found that making “none of the above” the correct answer caused a consistent 30–50% performance drop in frontier AI models. Griot and colleagues showed that models often reach the right multiple-choice letter through shallow cues and test-taking strategies rather than clinical reasoning – concluding that standard MCQ evaluations may not measure clinical reasoning at all.

On LiveMedBench, 84% of evaluated models performed worse on cases that postdate their training cutoff – strong evidence that earlier scores reflected memorization. Outside medicine, audits have found roughly 29% of MMLU items show contamination signs, with some models dropping double-digit percentage points on clean rewrites.

When a benchmark has been on the public internet for years, a strong score is partly a reading comprehension test of the model’s own training data. The authors of the Nature Medicine paper know this (they say so), which is exactly why their single private benchmark (RCQ) carries so much weight. And RCQ is 100 questions from one hospital, with no public description of how they were selected and no way for anyone else to reproduce the result.

The data nobody can crawl

Here is the asymmetry that I think the headline misses entirely. The tasks frontier models are good at – exam questions, textbook knowledge, summarizing the medical literature – sit on top of data that is abundant and public. Millions of journal articles, guidelines, and Q&A threads are openly crawlable. Of course general models trained on the whole public internet do well there.

The tasks that actually run a health system sit on top of data that is scarce and private. There is no large, public corpus showing how a radiology report should be summarized for a referring physician, how a referral letter should be written, or how tumor characteristics (histology, grade, stage, margins, receptor status) should be extracted from a pathology report and mapped to a registry standard. That knowledge lives inside electronic health records protected by HIPAA, institutional review boards, and data use agreements. You can see this constraint everywhere in the serious benchmarks: MEDEC’s hospital notes require a signed DUA; MedHELM’s clinical sources are largely private or gated; the Nature Medicine paper’s own RCQ data “is not available for public use due to institutional review and data use agreement.” A model can’t memorize what it was never allowed to see, which is most of clinical work.

The OpenEvidence episode

The study landed hardest on OpenEvidence, which responded publicly, on LinkedIn and in a June 15 letter to the journal requesting a retraction, alleging undisclosed conflicts of interest and methodological flaws.

Some of those points are reasonable, and several overlap with limitations the authors themselves listed. The contamination concern about MedQA and HealthBench is valid. The criticism that HealthBench’s scoring is opaque and built by a competitor is fair. And the observation that the RCQ dataset and its selection methodology were never published is correct. On the conflict of interest charge, the picture is murkier: the paper does disclose that the senior author consults for Google, whose Gemini topped every stage. So “undisclosed” is contestable, but it’s a relationship worth weighing when reading the result, just as funding source matters greatly in nutrition research.

It’s also worth remembering that OpenEvidence has used marketing-via-science of its own. In August 2025 it announced it was the first AI to score a perfect 100% on the USMLE. That number drew immediate skepticism: 100% on a saturated multiple-choice format that overlaps with MedQA says little about messy real-world care, and others noted the result was hard to reproduce. The honest read is that neither a vendor’s “100% on USMLE” press release nor a study titled “general models outperform clinical tools” tells you what you actually want to know.

The ranking isn’t even stable across tasks. On hard subspecialty board questions (the MedXpertQA set), a December 2025 pilot found OpenEvidence’s quick and “Deep Consult” modes scoring just 34% and 41%, below the roughly 46% the best general reasoning model reached on the same questions. Flip to open-ended, real-world clinical questions, though, and the order inverts: in a peer-reviewed study of 50 such questions, general-purpose LLMs produced relevant, evidence-based answers only 2–10% of the time, versus 24% for a retrieval-augmented system (OpenEvidence) and 58% for an agentic real-world-evidence system (ChatRWD). The winner depends entirely on the task, which is the whole reason a one-task study can’t crown a general one.

Back to first principles: fine-tuning to a task improves accuracy on it

Strip away the branding and this reduces to something every data scientist already knows: for any model, fine-tuning to a specific task or dataset raises accuracy on that task. That isn’t a marketing claim; it’s the bias-variance tradeoff. A model specialized for clinical information extraction, trained on high-quality data annotated by clinicians, will tend to beat a generalist on clinical information extraction – the same way the generalist beats it at writing a sonnet.

This is the work John Snow Labs does – de-identification, information extraction, and question answering over real clinical records – and the numbers reflect the principle rather than contradict it. On PHI detection our pipelines report 96% F1 versus Azure’s 91%, AWS’s 83%, and GPT-4o’s 79%, and 0.95 F1 against OpenAI’s privacy filter at 0.55, running 5.8× faster on CPU. Across a 13-task clinical benchmark suite, which overlaps heavily with MedHELM, our medical language models average 76.8 versus 70.9 (GPT-5.4), 70.0 (Gemini 3.1 Pro), and 68.3 (Claude Opus 4.6), ranking first on 12 of 13 tasks. The model that produces those scores runs on a single GPU, entirely inside the customer’s environment, with no external API call. On these problems, raw scale and trillion-token training budgets matter far less than domain data and careful engineering.

Today’s state-of-the-art results go further than that. By wrapping specialized models in an agentic feedback loop – generation, deterministic checking, iterative correction – we’ve reached regulatory-grade accuracy, meaning at or above human experts, on tasks like de-identification and patient registry abstraction. An early pipeline that de-identified 2 billion patient notes was externally certified at that bar, with recall exceeding independent human annotators. The newer version, an agentic de-identification framework we presented at this year’s Data + AI Summit, treats the pipeline itself as an agent that automatically tunes and customizes itself to a previously unseen dataset, reaching 98%+ regulatory-grade accuracy with minimal human effort.

On the documentation side, a cancer registry abstraction system – built from small language models, medical NLP, and a deterministic reasoning layer – cut case abstraction from about 120 minutes to under 2 while holding regulatory-grade accuracy, in a setting where 96% of incoming pathology reports are non-reportable noise. That kind of consistent, auditable, at-or-above-human accuracy at scale is not something a general-purpose LLM delivers on its own today.

What I hope we take from it

I’m glad this study exists. Specialized clinical tools should face independent, quantitative scrutiny, and the field needs far more of it. The authors did careful work and were honest about its limits.

It’s worth adding, though, that this kind of evaluation already runs in the open, continuously and not as a one-off. MedHELM is a free community service: an open-source extension of Stanford’s HELM framework (maintained by my team at Pacific AI), it re-scores the newest frontier models on a roughly quarterly cadence across its full clinician-validated taxonomy. The latest release folds in both MedQA and OpenAI’s HealthBench (Original and Professional) so the very benchmarks this paper relied on now sit alongside three dozen others, and anyone can rerun the whole suite and reproduce the numbers themselves. A single Brief Communication is a snapshot; a public, versioned, reproducible leaderboard is what actually keeps everyone honest over time.

My only real objection is to the headline – and to the reflex, on all sides, to compress a narrow result into a universal claim. Single-turn medical Q&A on partly-public benchmarks is one task. “Medicine” is a hundred. Good evaluation requires many real-world tasks, contamination-resistant test sets, real clinical data, and titles scoped to what was actually measured. Get the question right and the answer is rarely “general models win” or “specialized models win.” It’s “it depends on the task” – which is less of a headline, but better science, and a lot more useful to anyone deploying this technology in a hospital.