What we learned building Medical LLMs that academic medical centers trust

Originally published alongside the 2025 InfoWorld Technology of the Year Award announcement, December 2025.

InfoWorld named John Snow Labs’ Medical LLMs a 2025 Technology of the Year Award winner in December. The press release covered the recognition. This post covers what sits behind it: the engineering and evaluation choices we made, what we got wrong early on, and what the last three years of deploying medical language models inside health systems, pharma companies, and government agencies have taught us about what regulatory-grade domain AI actually requires.

The thesis we started with

When we began building Medical LLMs in earnest, the prevailing view was that frontier general-purpose models would absorb every vertical. The path for specialized AI, under that view, was either fine-tuning on top of a frontier model or retrieval-augmented generation layered on the frontier model’s reasoning. Specialized pretraining was considered either impossible (too expensive) or unnecessary (the gap would close).

We took the opposite bet. Healthcare is a domain where the vocabulary, the reasoning patterns, the regulatory environment, and the deployment constraints are different enough from general enterprise use that a purpose-built approach would outperform: on accuracy, on cost, and on the compliance properties that determine whether a model can run against PHI at all.

Three years later, that bet has paid out. Peer-reviewed benchmarks show Medical LLMs outperforming GPT-4.5 and Claude 3.7 by 61–200% in factuality, clinical relevance, and conciseness, while running at a fraction of the cost. The models are deployed at Providence St. Joseph Health, Roche, the VA, Ohio State, Cigna, and others, with 80+ public case studies describing real-world use.

But the thesis by itself wasn’t the hard part. The hard part was operationalizing what “regulatory-grade” actually has to mean.

What we got wrong first

The first generation of our medical LLMs focused on medical knowledge benchmarks: MedQA, PubMedQA, MMLU medical subsets, MedMCQA. We hit state-of-the-art numbers. We published the results. And we quickly learned that leaderboard accuracy on medical licensing exam questions did not predict real-world utility on the tasks clinicians and researchers actually needed.

Clinical information extraction from progress notes is not an exam question. Cancer registry abstraction from pathology reports is not a multiple-choice problem. Clinical summarization under time pressure with a 50-note patient history requires different capabilities than answering a textbook question correctly on the first try.

The lesson, which has now been documented in peer-reviewed work including the CLEVER paper out of Stanford and elsewhere, is that leaderboard performance and real-world performance correlate loosely in healthcare. CLEVER showed frontier LLMs dropping from 92% accuracy on benchmark questions to 45% on equivalent real-world clinical tasks. We saw similar patterns in our own evaluations.

The corrective was to reorganize evaluation around the tasks that customers actually deploy: clinical entity extraction with terminology mapping, clinical summarization scored by clinician preference, question answering over real patient records, assertion and negation handling on free-text notes, temporality reasoning across longitudinal data. Benchmark numbers still matter as a necessary filter, but they are not a substitute for evaluation that looks like the deployment.

What “medical LLM” actually means in practice

A healthcare-specific LLM is more than a frontier model with a medical finetune on top. The differences show up at four architectural layers.

Pretraining data composition matters more than parameter count. Our Medical LLMs were trained on curated biomedical literature, clinical guidelines, de-identified EHR notes, and life sciences content. The diversity of clinical documentation types (discharge summaries, operative notes, radiology reports, pathology reports, nursing notes) teaches the model the style variance that a clinical deployment actually sees. Training on Reddit and Common Crawl does not.

Model sizing is a design variable, not a scaling axis. We ship 7B, 13B, and 70B variants, plus specialized reasoning, visual, and Spanish-language models. The reason is that clinical deployment cost economics don’t match consumer chatbot economics. A health system processing millions of notes per day cannot afford a 400B-parameter model per query when a 13B model with appropriate training produces better output for their specific task. Peer-reviewed benchmarking showed our 7B model outperforming all prior 7B models on clinical tasks, and becoming the first 7B model to beat GPT-4 on PubMedQA.

Alignment and safety training have to include clinical failure modes. The failure patterns of a medical LLM are not the same as those of a general chatbot. Hallucinating a medication dose is not the same kind of error as hallucinating a movie title. Our alignment work focused specifically on clinical hallucination, incorrect differential reasoning, dangerous recommendations, and the sycophancy patterns that lead models to agree with an incorrect clinical premise from the user. The Pacific AI governance work, including MedHELM, LangTest, and red-teaming frameworks, extends this to ongoing testing in deployment.

The deployment architecture is part of the product. Medical LLMs that can only be accessed via API are incompatible with most healthcare deployment requirements. Our models run inside the customer’s environment: on-premises, in private cloud tenants on AWS, Azure, GCP, Databricks, or Snowflake, or in air-gapped government environments. PHI cannot leave the customer’s firewall, and because HIPAA, GDPR, and sector-specific data agreements don’t allow it. Model quality is necessary. Compliant deployment is equally necessary. Models that can’t run where the data lives are not useful in production.

What the InfoWorld award actually recognized

InfoWorld’s judging notes described the models as “advanced domain-specific models with large context windows, multimodal capabilities and benchmark results that show strong performance,” with explicit recognition of privacy and compliance requirements. That framing matches what the last three years have taught us: domain specialization, deployment architecture, and compliance properties are not separable features. They are a single design choice.

Customers implementing Medical LLMs report 80% less manual abstraction, go-live in under two weeks, and around 60% lower operating costs compared to API-based alternatives. Those numbers come from the same underlying architecture: healthcare-specific pretraining, right-sized model footprints, in-environment deployment, and human-in-the-loop workflows via the Generative AI Lab.

What we’re focused on next

Three capability directions are absorbing most of our engineering effort in 2026.

Multi-agent architectures for clinical workflows. Single-model reasoning works for extraction and summarization. Multi-step clinical workflows (cancer registry abstraction, HCC coding review, clinical trial matching, patient journey reconciliation) benefit from specialized agents coordinated by a planner. The architecture is converging on domain-specific agents that hand off work to each other under a governance layer that logs provenance and enforces compliance controls.

Multimodal clinical understanding. Clinical reality is multimodal: text, structured data, imaging, video, waveforms. Our Visual LLM handles pathology slides, radiology images, and clinical PDFs. The direction is toward unified reasoning across modalities, with the same accuracy, provenance, and compliance properties we deliver on text.

Agentic data pipelines for OMOP and FHIR. The Patient Journey Intelligence platform we launched in early 2026 integrates multimodal, longitudinal clinical data into unified OMOP data models. The engineering work underneath is agentic: pipelines that read source documents, resolve identities, reconcile conflicting records, map to terminologies, and produce FDA-ready real-world evidence datasets. The requirements are the same as the Medical LLM work (regulatory-grade accuracy, in-environment deployment, auditable provenance), applied one layer up.

What the award means

Awards are a useful external signal. They are not the work. What the InfoWorld recognition reflects is three years of compounding engineering choices: pretraining on the right data, sizing the models for the deployments they’ll actually run in, aligning them for clinical failure modes, and packaging them so they can run inside the customer’s security perimeter.

The healthcare AI market is still early, and the gap between what general-purpose AI can do and what regulated clinical deployment requires is still wide. Closing that gap, while keeping the accuracy, compliance, and cost properties that health systems and life sciences organizations need in production, is the work we’re signed up for.

Frequently asked questions

What is a Medical LLM and how is it different from a general-purpose LLM?

A Medical LLM is a large language model trained specifically for clinical and biomedical use. The differences from a general-purpose LLM are in the pretraining data (curated biomedical literature, clinical guidelines, de-identified EHR notes, life sciences content), the alignment and safety training (focused on clinical failure modes), the deployment architecture (runs inside the customer’s firewall, not via API), and the task performance (peer-reviewed benchmarks show domain-specific models outperforming frontier LLMs by 61–200% on factuality and clinical relevance).

Why does a smaller specialized model often beat a larger frontier model on clinical tasks?

Three reasons. First, training data composition matters more than parameter count on domain-specific tasks; a 7B model trained on high-quality clinical content outperforms a much larger model trained on general web data. Second, clinical tasks reward specific capabilities (negation, temporality, terminology mapping) that domain pretraining teaches and general pretraining doesn’t emphasize. Third, cost economics in clinical deployment favor smaller models because volume per customer is high and latency matters.

Do Medical LLMs make clinical decisions?

No. They extract information, summarize records, answer questions against existing documentation, and support clinician and researcher workflows. Clinical decisions remain with clinicians. The regulatory path for decision-support AI is different from the path for clinical data extraction and summarization, and the accuracy and validation requirements differ accordingly.

How do Medical LLMs handle hallucination?

Several ways. Training on curated clinical content reduces baseline hallucination rates. Retrieval grounding against the patient’s actual record provides source evidence for every extracted claim. Assertion and provenance tracking attaches each extracted entity to the source sentence that supports it. Evaluation frameworks like MedHELM and LangTest test for hallucination specifically on clinical tasks. Human-in-the-loop review via Generative AI Lab is the final check on any output that reaches a clinical decision point.

Where can Medical LLMs be deployed?

On-premises, in private cloud tenants on AWS, Azure (via the Azure Marketplace), GCP, in Databricks and Snowflake environments, and in air-gapped environments for government and defense customers. The deployment pattern keeps PHI inside the customer’s environment throughout processing: no external API calls, no data leaving the firewall.

What makes the evaluation of Medical LLMs different from general LLMs?

Task-specific, clinician-validated evaluation matters more than standardized benchmarks. Benchmarks like MedQA or MMLU are necessary filters but not sufficient. The CLEVER paper out of Stanford showed frontier LLMs dropping from 92% accuracy on benchmarks to 45% on equivalent real-world clinical tasks. Evaluation has to include clinical information extraction, summarization scored by clinician preference, and blind comparison against expert annotations on real patient records.

Why does the architecture include a no-code review tool?

Because the people who need to validate and improve clinical AI are often domain experts (coders, registrars, clinicians, RWE analysts) rather than ML engineers. The Generative AI Lab lets them evaluate model output, correct errors, and continuously improve the pipeline without writing code. That closes the feedback loop much faster than a traditional ML workflow would, and it’s essential for the human-in-the-loop patterns that regulatory-grade deployment requires.