Small, Private, and First on All Fifteen: The New Medical LLM Benchmark Results
First on all 15 clinical and biomedical benchmarks, averaging 80.9 against the newest frontier releases, running on a single GPU inside your own environment.
John Snow Labs’ latest Medical LLM results are in. Across fifteen clinical and biomedical benchmarks, the Medical LLM – Medium ranks first on every one, against the newest frontier releases from OpenAI, Anthropic, and Google. It averages 80.9 to their 76.5, 75.0, and 74.7 respectively. The model that produces those scores runs on a single GPU, entirely inside your own environment, with no external API call. It is licensed per server per year, not per token.
A leaderboard sweep is not worth much on its own, but this combination of accuracy, cost, and compliance is what deserves attention. It contradicts an assumption that governs most healthcare AI budgets right now: that the accurate option and the affordable, deployable option are different options, and you have to pick one.
The suite comes from MedHELM, not licensing exams
A benchmark suite assembled only from USMLE-style questions and PubMed recall measures one capability and calls it medicine. Most of these 15 tasks therefore come from or overlap with MedHELM, the Stanford-led open benchmark published in Nature Medicine, which organizes clinical AI into a clinician-validated taxonomy of 121 distinct tasks. Each selected task asks a model to do something a health system needs done.
Medec gives the model a clinical note and asks whether a sentence contains an error, such as a wrong diagnosis, a wrong drug, or a wrong causal organism, then asks it to point to the sentence and rewrite it. This is the inverse of a quiz: the note is written to look correct, and the model has to disagree with it. The best models scored around 70% on the original benchmark, against roughly 80% for physicians.
MedCalc-Bench asks for a clinical calculation from a patient vignette. Computing a Cockcroft-Gault creatinine clearance means pulling the right values from the right encounter and applying the right formula. It is the hardest task in the set, and the absolute scores of 48.0 against 34.0 for Claude Opus 4.8 show unsolved work rather than a lead.
EHRSQL turns a clinician’s plain-English question into SQL over a hospital database, along the lines of “how many patients were prescribed warfarin in the last month.” Built from questions posed by 200+ real hospital staff, it includes unanswerable questions on purpose, to test whether a model abstains instead of inventing a query. Gemini 3.5 Flash scores 14.0 against 34.0, the widest proportional gap in the table.
ACI-Bench takes the raw transcript of a doctor-patient conversation and asks for a structured visit note. The input has interruptions, repetition, and facts stated out of order. The output is documentation a clinician signs.
MTSamples Procedures asks the model to find and extract the surgical and medical procedures buried in a transcribed report. Procedures are rarely announced in a labeled field, and one report will mention procedures performed, considered, declined, or done elsewhere years earlier. Scoring well means separating what was done from what was discussed.
MedDialog scores comprehension of patient-provider conversation, measuring whether a model follows what a patient said rather than a nearby topic. At 76.3 against 76.2, this row is a four-way tie in everything but the decimal.
MedicationQA uses real consumer medication questions submitted to a health authority, rather than questions written to be answerable. The 9.5-point margin over the best frontier model is the second-widest in the suite.
The remaining tasks cover knowledge and reasoning (HeadQA, MedBullets, PubMedQA, MEDIQA, MedQA, MMLU Clinical Knowledge), hallucination control (Med-Hallu), and fairness across demographics (RaceBias).
Gaps are near zero on saturated exams and widest on operational tasks
Two patterns emerge from this table, the first coming from the medical exam rows. On MMLU Clinical Knowledge the spread from top to bottom is 2 points. On MedQA the margin is 1.2 points, and Gemini ties GPT outright. These benchmarks are saturated: everyone scores in the 90s, the differences approach noise, and a strong score is partly a reading comprehension test of the model’s own training data, as our previous discussion on data contamination detailed. No healthcare-specific model opens a real lead on a saturated public benchmark, and none should claim to.
The second pattern comes from the operational tasks, where the margins are an order of magnitude larger: 15 points on Medec, 9.5 on MedicationQA, 6 on PubMedQA, 5 on EHRSQL and RaceBias, 4 on MedCalc and Med-Hallu. The closer a task gets to clinical operations, meaning read a messy source document, extract the right facts, do the arithmetic, catch the error, and know when the answer isn’t supported, the wider the margin gets.
The saturated exam benchmarks stay in the suite so both patterns remain visible. Publishing only the wide margins would tell you less.
Private deployment costs an order of magnitude less at production volume
Accuracy decides whether a pipeline is worth running. Cost decides whether it survives a budget cycle. A node that is paid for processes its millionth token at the same marginal cost as its first: zero.
I modeled this last month on a real workload: a de-identified oncology real-world evidence dataset at 1 million tokens per patient, four passes over every document. The model uses list API prices, grants every volume discount, and skips the document filtering optimization that would favor local deployment. What frontier APIs cost, relative to a right-sized model running locally:
• 10,000 patients (a pilot or one service line): 1.35x to 3.17x. At this scale the API route is competitive, and for a short proof of concept the premium can be rational.
• 100,000 patients (a small health system or a focused research cohort): 3.3x to 7.7x.
• 1,000,000 patients (a midsize health system, a payer, or a multi-site research network): 12.3x to 28.8x.
Per-token pricing makes cost linear in data volume, while infrastructure grows sublinearly: a 100x increase in patients needed 15x the nodes, and cost per node fell by roughly a quarter along the way. This protects small players rather than threatening them. A pilot costs about the same either way, and the deployment that stays affordable is the one that doesn’t re-price every time the data grows.
The behavioral consequence matters more than the invoice. Teams ration when the marginal token costs money: discharge summaries get processed and nursing notes skipped; two years get reviewed instead of the full patient history. Every one of those decisions degrades the output. When the marginal token is free, the rational behavior inverts: process every page of every record, and rerun whenever the pipeline improves.
A general-purpose frontier model bills you for capabilities a pathology report never uses
“Small model beats big model” sounds like a result the next frontier release erases. But the reason it persists is structural.
A general-purpose frontier model is general-purpose by obligation. The same weights that read a discharge summary also have to translate poetry from Sanskrit, name the street corner in an uploaded photograph, and assemble a timeline of Russian philosophy from memory. That breadth is the product, and it is why the model is the size it is.
None of it reaches a pathology report. Being able to write Sanskrit poetry contributes nothing to tumor stage extraction, and remembering the whole public internet in every language contributes nothing to reconciling a medication list. You pay for that storage and compute regardless, on every token, whether you rent the model through an API or install it yourself. No engineering team would provision a database where 99% of the tables are irrelevant to all queries. Frontier LLMs are the one place the industry accepts it without argument.
That arithmetic is what makes “deploy your own LLM” sound unreasonably expensive. The phrase brings to mind running a frontier-scale model yourself, with the hardware and specialist team that requires. That premise is wrong. A frontier-scale model was never the requirement. A model that is very good at your tasks is.
Right-sizing inverts the cost structure. Medical LLMs are purpose-built for medical language rather than general text, trained on curated biomedical literature, clinical guidelines, and de-identified EHR notes. Sized for the tasks they run, they fit on one GPU, which makes them cheap enough for population scale and portable enough to deploy on-premises, in a private cloud tenant, or in an air-gapped environment.
Specialization raises accuracy on those tasks rather than trading it away. Training-data composition matters more than parameter count on domain work, and the tasks that separate the models above depend on clinical documentation that is scarce and private. Accuracy, cost, and compliance therefore come out of a single design choice rather than arriving as three separate features. HIPAA, GDPR, and institutional data agreements keep PHI inside your environment, as well as your intellectual property.
What the 15-benchmark sweep does not show
Healthcare-specific models do not win everything. On open-ended, single-turn general medical question answering, frontier models are excellent, and the saturated rows above show gaps within noise. A clinician typing a general question into a chat box is well served by a frontier model.
Benchmarks are also not the same as production deployments. John Snow Labs’ first-generation medical models hit state-of-the-art numbers on MedQA and PubMedQA, then underperformed on the extraction work customers ran in production. That history is why this suite looks the way it does. The CLEVER framework, published in JMIR AI, found frontier LLMs dropping from 92% on benchmark questions to 45% on equivalent real-world tasks. Benchmarks are a necessary filter and never a substitute for blind clinician review on real charts.
Nor are these numbers frozen. MedHELM is open source, maintained by Pacific AI as an extension of Stanford’s HELM framework, and re-scored against the newest frontier models on a roughly quarterly cadence. A vendor’s table is a claim but a public, versioned, reproducible leaderboard is a claim anyone can check.
Nine years of shipping every two weeks
John Snow Labs has released new software every two weeks for nine years. Through several generations of technology and science, from feature-based NLP to word embeddings to BERT-era transformers to instruction-tuned LLMs to agentic pipelines. Every one of those transitions obsoleted work the team was proud of.
The Medical LLMs have improved on nearly every release over the past year. The spring comparison covered 13 benchmarks against GPT-5.4, Gemini 3.1 Pro, and Claude Opus 4.6, ranking first on 12 of them. This one covers 15 tasks against the newest releases from all three labs, ranking first on 15. The suite grew, the competition got stronger – yet the margin widened.
That cadence holds because John Snow Labs is profitable, independent, and built without a cent of venture or debt financing. Nobody is waiting on an exit. A two-week release cycle is only worth something if it’s still running in five years, and the companies that can promise that are the ones with no clock on them.
The choice between accuracy, cost, and privacy was never permanent
Healthcare AI has spent three years budgeting as though you must choose two of three: accuracy, affordability, or privacy. Frontier APIs offered accuracy and convenience, and asked you to send PHI elsewhere and pay per token forever. Local models offered privacy and cost control, and asked you to accept a worse model. That trade-off is no longer true.
Within a domain that has its own language, its own documentation conventions, and training data that never reaches the public internet, the specialized model is now the accurate one, and it is small enough to run where the data already lives. Healthcare is the clearest case rather than the only one. Any field with a private vocabulary and a compliance perimeter should expect the same result, and should stop planning around a tradeoff that no longer binds.
Frequently asked questions
What is the Medical LLM – Medium?
It is John Snow Labs’ healthcare-specific large language model, trained on curated biomedical literature, clinical guidelines, and de-identified EHR data, and sized to run on a single GPU inside your environment. It is licensed per server per year, with no per-token component and no external API dependency.
Why not evaluate on USMLE-style questions alone?
Because they measure one capability while the field reads them as measuring medicine, and because they are near-saturated and partly contaminated. A near-perfect exam score says little about whether a model can extract a tumor stage from a pathology report, catch a medication error in a note, or abstain from a question the record can’t answer.
How does a single-GPU model beat much larger frontier models?
Training-data composition matters more than parameter count on domain-specific tasks. Clinical work rewards negation, temporality, terminology mapping, calibrated abstention, and faithfulness to a source document, which domain pretraining teaches and general pretraining doesn’t emphasize. The operational tasks also depend on private clinical data that public-web scale never substitutes for.
Is running a private medical LLM expensive?
Only if a frontier-scale model is assumed to be the requirement. At pilot volume the difference against APIs runs 1.35x to 3.17x and rarely decides anything. At a million patients, modeled on an oncology real-world evidence workload, frontier APIs cost 12x to 29x more, because per-token pricing is linear in data volume while infrastructure is not.
Do Medical LLMs make clinical decisions?
No. They extract information, summarize records, answer questions against existing documentation, and support clinician and researcher workflows. Clinical decisions stay with clinicians.