Every clinical assertion in a regulatory-grade healthcare AI platform must preserve its source document, the exact span it came from, the model and configuration that extracted it, the confidence assigned at extraction, the conflicts detected against other sources, and the rule or reviewer that resolved them. That is fact-level provenance. Building it takes 42 distinct capabilities across ingestion, extraction, privacy, reasoning, audit, and versioning. Retrofitting it onto an existing warehouse is far harder than carrying it through from the first parse.
The FDA’s December 2025 final guidance on real-world evidence for medical devices, operational since February 2026, treats relevance and reliability as per-fact properties of a submission, not per-dataset attributes. When a reviewer asks where a value came from, the answer should be given in a click, not a forensic project.
The clinical fact as the unit of governance
Most data warehouses govern at the level of the file or the table: access controls per dataset, versioned snapshots, dataset-level data dictionaries. That granularity is sufficient when the regulatory question is about a study population. It is not sufficient when the question is about a specific value for a specific patient.
When a reviewer asks “where did this come from?” about a single comorbidity, date, or medication dose, an auditable answer has to include the source document, the exact span in that document where the value appeared, the model and prompt version used to extract it, the confidence assigned at extraction, the normalization decisions that followed, the conflicts detected with other sources, and the rule or human reviewer that resolved them. That is the unit of governance.
Every clinical fact in the system carries six categories of attributes:
These attributes are native columns on the fact record, not entries in a separate lineage table, and they propagate into every derived measure, cohort, and agent answer. Once facts are stored without them, adding them requires re-processing the full corpus against models and documents that may no longer exist in the same state. That is the design decision the rest of the architecture rests on, and it must be made before the first data pipeline runs.
Three re-derivable tiers from raw bytes to OMOP
A monolithic pipeline that parses, extracts, normalizes, and reasons in one pass is the most natural thing to build. It is also impossible to reproduce months later. The version of the parser, the prompt to the extraction model, the terminology mapping table, the conflict resolution rule: all of these change. Without a structural separation between layers, point-in-time reproduction means standing up the entire stack at its earlier state, which is rarely feasible. Instead, the pattern that holds up under audit is a tiered one.
Bronze: lossless parsing. Every file and message format ingested without information loss. Free-text notes, FHIR R4 resources, HL7 v2 messages, DICOM headers, scanned PDFs with OCR, structured warehouse extracts. Nothing dropped, nothing normalized. This tier is immutable: it records exactly what arrived. When a downstream extraction model is retrained two years later, you can re-run extraction against the original Bronze records and reproduce or improve on the earlier result.
Silver: extraction with provenance. Clinical facts pulled from every modality, tagged with source coordinates, scored for extraction confidence, and mapped to standard terminologies. Each fact is independently traceable to a Bronze record. Silver can be re-derived from Bronze without touching anything upstream.
Gold: reasoning and standardization. Duplicates merged, conflicts reconciled across documents, measures and risk scores computed, and the result emitted in a shared analytic format, most commonly OMOP CDM or FHIR. Gold can be re-derived from Silver, and ultimately from Bronze.
Each tier can be validated, re-run, and audited independently. From any value in Gold, an auditor can walk all the way down to the raw bytes. The 21 CFR Part 11 expectation of reproducibility, first codified in 1997 for electronic records and now extended into a per-fact requirement under the new FDA RWE guidance, only holds when every layer is versioned and the dependencies between layers are explicit.
The 42-capability inventory
No single component in this inventory is exotic. The difficulty is that the platform needs 42 distinct capabilities, which must be planned from the first day, and which interact with one another to add complexity. Grouped by domain:
Multimodal ingestion (Bronze)
Free-text clinical-note parsing with structure preservation (section headers, list semantics, table extraction)
FHIR R4/R5 resource ingestion with reference traversal
HL7 v2 message parsing with segment and field-level access
DICOM header parsing across all standard SOP classes
DICOM pixel-level PHI detection for text burned into images
PDF text extraction for both digital and OCR-required scanned documents
Connector framework for SQL warehouses, claims feeds, and CSV imports
An immutable Bronze record store with content-addressed identifiers
Clinical extraction (Silver)
Healthcare-specific named entity recognition across condition, procedure, medication, lab, and vital domains
Relation extraction linking entities that only carry meaning together: a drug to its dose and frequency, a tumor to its stage and site, a lab value to the panel it belongs to
Assertion status detection (confirmed, ruled-out, family history, patient-reported, historical)
Negation and uncertainty detection
Temporality detection and date normalization across relative (“last year”) and absolute references
Terminology resolution to SNOMED CT, RxNorm, LOINC, ICD-10-CM, and CPT
Per-fact extraction confidence scoring, exposed as a real-valued attribute on the fact
Per-fact source coordinates: document ID plus character span, DICOM tag path, or FHIR JSON pointer
The accuracy bar for clinical extraction is high enough that general-purpose LLMs underperform. A peer-reviewed head-to-head evaluation we published at ECIR 2025 found a healthcare-specific small model reaching 96% F1 on PHI detection, against 79% for GPT-4o, 83% for AWS Comprehend Medical, and 91% for Azure, at over 80% lower cost.
The cost economics matter for capability #15 in particular: confidence scoring runs on every fact in the corpus, so a token-billed API call per fact is non-viable at population scale. The architectural choice that follows is to use specialized small models for the billions of routine extraction decisions, and reserve large generative models for narrative generation and multi-step reasoning where their breadth earns its place.
Privacy and de-identification
PHI detection across the 18 HIPAA Safe Harbor identifier categories in free text
PHI detection in PDF, combining text-layer and OCR coverage
PHI detection in DICOM headers
PHI detection in DICOM pixels (burned-in text), using vision-language models
Consistent pseudonymization across documents and modalities, preserving longitudinal linkage
Patient-specific date shifting that preserves intra-patient temporal relationships while preventing date-correlation re-identification across systems
Parallel maintenance of identified and de-identified datasets, kept in sync by the same ingestion pipeline
Configurable de-identification profiles (HIPAA Safe Harbor, Expert Determination, GDPR pseudonymization)
The parallel-dataset choice in capability #23 deserves a closer look, because the intuitive design fails quietly. That design stores everything identified and de-identifies on export. That design fails the GDPR Article 25 “data protection by default” test, and it fails the HIPAA Minimum Necessary standard for almost every research workload. The structural design maintains both datasets continuously, defaults every secondary-use query to the de-identified one, and requires explicit elevated permission for any read against the identified one. Privacy then operates as a property of the schema and the query path, enforced by the platform itself.
Reasoning and reconciliation
Patient-level record linkage across source systems
Cross-document deduplication of clinical events
Conflict detection when sources disagree (chart says 80mg, pharmacy feed says 40mg)
Temporal reasoning to distinguish “history of diabetes” from “current diabetes”
Implied inference rules with explicit confidence (patient started metformin → likely but not confirmed diabetic)
Absence-as-negative inference, scoped to documents that would plausibly mention the finding (a cardiology note that does not mention HIV is not evidence; an HIV screening result that does not mention HIV is)
Decay rules for time-sensitive measurements (a five-month-old weight, a one-hour-old pulse)
Extraction is the visible engineering. Reconciliation is where the platform either picks a value silently and accumulates quiet errors, or surfaces the conflict with its evidence and routes the decision somewhere it can be audited. The auditable design picks the second path, which requires a reasoning layer with explicit policies for every type of conflict the data may produce. That layer is months of work even after the extraction is good.
Audit and access control
Tamper-evident audit logs capturing user identity, accessed records, timestamp with millisecond precision, purpose code, and query text, for every access path
Role-based access control over every data asset and every column
Purpose limitation enforced at the database layer, so a query outside the project’s authorized scope is rejected regardless of the SQL
Consent tracking integrated with the patient record, enforced at query time against current consent status
Anomaly detection on access patterns (10,000-patient reads against historical baselines under 100)
The audit log entry that supports this captures, for every access event:
The hash chain in the last row is what makes the log “tamper-evident” in regulatory parlance. An administrator with database access can append, but cannot rewrite history without breaking the chain. That property is required for regulated submissions and useful in its own right when answering an audit question months after the fact.
Versioning and reproducibility
Immutable dataset versioning at all three tiers
Model version pinning with content-addressed artifacts, covering every language model in the pipeline: the specialized NLP models doing entity recognition, assertion detection, and de-identification, and the LLMs doing reasoning and generation
Extraction configuration versioning: the full pipeline definition active for every run, including the sequence of stages, the prompts, the guardrails, and the decoding parameters
Terminology mapping versioning (SNOMED CT, RxNorm, LOINC, and ICD-10-CM each release on their own cycles, from monthly to annual)
Business rule versioning for the reasoning layer, so a change to which source wins a medication conflict creates a new version without invalidating prior runs
Point-in-time dataset reconstruction: given a query and a timestamp, return the result the system would have returned at that time
Capabilities #38 and #39 are where the choice of an external LLM API grows from a cost question into an architecture question. A hosted frontier model is not deterministic: the same prompt against the same endpoint returns different outputs across runs, and even at temperature zero, providers do not guarantee identical results. It is not pinnable either. The model behind an API endpoint is updated on the provider’s schedule, versions are deprecated and retired on cycles measured in months, and a version retired in 2026 cannot be re-run in 2028 no matter what your audit requires. An extraction step you cannot re-run against the same model, prompt, guardrails, and decoding parameters is an extraction step you cannot defend, and the FDA’s per-fact reliability expectation makes that a submission problem, not a preference.
Capability #42 is the cleanest test of whether the rest of the work was done right. If you can ask “what would this cohort have looked like six months ago?” and get a deterministic answer in seconds, using the datasets, model versions, terminology releases, and business rules in force at that time, the versioning is genuine. If you cannot, somewhere upstream a version pin was missed. Content-addressed model artifacts on infrastructure you control pass this test. An API subscription does not.
That is the inventory. The engineering estimate, with a senior team that has built data platforms before but not this kind of healthcare-specific platform, is two to three years to land all 42 capabilities in a way that holds up under audit. With a team newer to the domain, longer. The capabilities are not individually exotic, but they compound a data platform’s complexity, and several of them (#23 parallel datasets, #16 source coordinates, #42 point-in-time reconstruction) only work if they were planned from day one.
The ten-step build order
Forty-two capabilities do not get built at once. The sequence below is ordered by what compounds and what retrofits worst.
1. Pick a shared analytic data model before the first pipeline runs. Two open standards cover most of the workload, optimized for different questions.
OMOP CDM v5.4, maintained by the OHDSI community, is the strongest default for secondary use: cohort definition, population analytics, real-world evidence, registry abstraction, and any question that compares groups of patients. OMOP is open, peer-reviewed, used at hundreds of institutions, and compatible with the OHDSI tool ecosystem (ATLAS, Achilles, HADES). Published RWE work is largely against OMOP, which makes reproducing prior results and contributing back tractable.
HL7 FHIR R4/R5 is the strongest default for single-patient analysis: clinical decision support, point-of-care AI, patient-specific question answering, and any workload where the unit of work is one patient’s record. The patient-centric resource graph makes joining one patient’s encounters, observations, conditions, and medications fast and direct, and every modern EHR speaks FHIR natively.
The choice is not exclusive. A platform that emits extracted facts into both formats covers the full secondary-use scope without forcing analysts into the wrong tool: FHIR for the per-patient agents, OMOP for the cohort and population work. What holds constant either way: pick the model before the first pipeline runs. A proprietary or homegrown schema locks every downstream analysis to one stack and forecloses on the published literature.
2. Build Bronze before extraction. The first sprint is the immutable raw layer, not the first extraction model. Capabilities #1 through #8 land first. Any extraction model you ship will eventually be replaced, and Bronze is what makes the next version re-derivable.
3. Make provenance a column type, not a metadata table. Source document ID, source span, extraction model version, and confidence score are attributes on every fact. This is the design choice that retrofits worst. Get it right at the first extraction.
4. Build de-identification as a pipeline stage, with parallel datasets from day one. Capabilities #17 through #24. De-identification built as an export-time step stops working once the second downstream consumer exists: someone, somewhere, will query the identified store for a research use case that should have hit the de-identified one.
5. Use specialized models for routine extraction, and reserve large LLMs for reasoning. A 96% F1 specialized model that runs deterministically on hardware you control beats a 79% F1 frontier API that costs more, returns different values run-to-run, and will be retired before your first re-audit. Match the tool to the task at every layer.
6. Build the reasoning layer as an explicit stage between Silver and Gold. Capabilities #25 through #31. Folding reconciliation into extraction makes the resolution policies invisible; folding it into the Gold materialization makes them unauditable. Reconciliation has its own policies, its own audit requirements, and its own confidence outputs, and it needs to be inspectable in isolation.
7. Version everything from the start. Capabilities #37 through #42. Versioning is the capability most likely to be “added later” and least likely to actually be added later, because by then there is too much un-pinned state to recover. Pin models, prompts, guardrails, pipeline configurations, terminology releases, and reasoning rules from sprint one, even when the pinning feels premature.
8. Treat human-in-the-loop as infrastructure. Conflicts the system cannot resolve confidently need a routing layer and a review UI that shows side-by-side evidence, records the human decision in the audit trail, and feeds the decision back as training signal. NAACCR cancer registry sign-off, NCDB abstraction review, and similar regulatory workflows already require this. The platform that bolts on a review screen at the end never matches the platform that designed the review queue into the pipeline.
9. One governed boundary for all agents. The Model Context Protocol (MCP) endpoint pattern is the cleanest version of this: agents call high-level platform tools (search_concepts, build_cohort, get_patient_timeline), not the underlying SQL tables. Redaction, masking, and access control apply inside the boundary, before any data leaves. Adding the tenth agent does not add a tenth governance surface; the governance is centralized once and inherited.
10. Continuous re-evaluation, not one-time audit. SNOMED CT and RxNorm ship new releases throughout the year, FDA guidance updates, state privacy laws change, specialized models drift on shifting documentation patterns. The platform that re-evaluates every dataset against current policy and current models, on a schedule, is the one that stays audit-ready. The platform that audits once a year is functionally not audited.
Patient Journey Intelligence and the FDA RWE guidance
Everything above reads as generic software architecture, and it is. It also describes an existing platform: Patient Journey Intelligence (PJI), John Snow Labs’ secondary-use data platform. It implements this pattern end to end: the six fact-level attribute categories as native columns, the three governance tiers, parallel identified and de-identified datasets from first ingestion, the hash-chained audit log, and four independently versioned layers (datasets, models and prompts, terminology releases, business rules) with point-in-time reconstruction as a first-class query. The data governance documentation walks through each layer in the same terms used here.
In January, John Snow Labs announced that PJI is the first secondary-use data platform designed to meet the FDA’s December 2025 final RWE guidance. The guidance makes two demands that map directly onto this article. First, clinical facts must be sufficiently complete and accurate, which means structured EHR data alone no longer suffices: unstructured clinical narratives, multimodal data, and longitudinal patient experience are in scope, because much of the clinically relevant information exists only there. Second, data provenance, quality, and reliability must be rigorously documented. PJI’s answer to both is the architecture above: every derived clinical fact carries end-to-end lineage back to its source document and exact location, versioned models, prompts, and rules from extraction, human-in-the-loop validation for high-stakes clinical endpoints, and deterministic reproducibility with confidence scores and full provenance.
The platform deploys entirely inside the customer’s infrastructure, with John Snow Labs’ healthcare-specific LLMs and SLMs included, so there are no external LLM API calls to break reproducibility and no PHI leaving the security perimeter. The de-identification pipeline inside it is the one validated across 2 billion patient notes with zero re-identifications. Model-level governance, meaning the registry, pre-release test gating, and production drift monitoring for every model in the platform, runs through Pacific AI, the CHAI-certified governance platform I also lead, deployed in the same tenant.
Six areas of ongoing work
Every architecture choice requires a tradeoff, and while building PJI we’ve encountered six areas that are still not fully solved. We are making progress on them every week, and we’re interested in learning from you if you work on these:
Extraction is not perfect. Even peer-reviewed regulatory-grade models miss and mis-assign facts. The pipeline reduces error and makes it visible through confidence scoring; it does not eliminate it.
Garbage in, governed garbage out. Provenance proves where a fact came from. It does not prove the source was correct. A confidently-wrong clinical note becomes a confidently-traced record. The architecture makes the error findable and fixable, which is the right bar, but it does not make errors disappear.
Specialized models need maintenance. Per-domain small models for SDoH, oncology, mental health, and the like must be re-validated as guidelines, terminology releases, and documentation patterns drift. The model catalog is a living cost, not a one-time build.
Human review is a real throughput constraint. Routing low-confidence fields to experts protects quality. Expert time is finite, the queue is genuine, and the platform that does not measure and manage review throughput will silently degrade.
OMOP cannot represent everything. Some clinical nuance (“adequate organ function,” “investigator believes the patient can comply”) resists any structured model. Forcing it loses meaning; leaving it out loses completeness. A design that holds up surfaces what cannot be represented and routes it to human judgment, rather than hiding the gap.
Evaluation itself is hard. Gold-standard labels for clinical extraction are scarce and expensive. Accuracy claims are only as strong as the reference standard behind them, and the reference standards in healthcare are smaller and noisier than the ones in general NLP.
42 capabilities and three years of engineering
The companies that move smoothly under the new FDA RWE guidance, and under the regulatory tightening that is going to keep arriving in 2026 and 2027, are the ones that made the architectural commitment to fact-level provenance early. Governance recorded only in documentation cannot reproduce a fact’s lineage on demand, and an audit is where that difference surfaces.
These forty-two capabilities are the scope of the clinical data platform you should build — whether you are a health system, a payer, a real-world evidence provider, or a data aggregator — and whether you build it or buy it. The design patterns are tiered ingestion, fact-level provenance, parallel identified and de-identified datasets, specialized extraction with confidence propagation, an explicit reasoning layer, versioning of everything including the models themselves, human-in-the-loop as infrastructure, and a centralized governance boundary for agents. They are what the platforms that survive AI governance reviews and regulatory submissions look like. Build to them on the first try if you can, or buy them already built. Budget for the rebuild if you can’t.
Frequently asked questions
What is fact-level provenance in healthcare AI?
Fact-level provenance means every clinical assertion in a data platform carries its own source document, exact source span, extraction model and configuration version, confidence score, conflict record, and resolution history as native attributes. Dataset-level controls can answer questions about a study population; only fact-level provenance can answer an auditor’s question about where a specific value for a specific patient came from.
What does the FDA’s December 2025 real-world evidence guidance require?
The final guidance on using real-world evidence to support regulatory decision-making for medical devices requires that clinical facts be sufficiently complete and accurate, which the agency recognizes structured EHR data alone cannot deliver, and that data provenance, quality, and reliability be rigorously documented. In practice, relevance and reliability become per-fact properties of a submission rather than per-dataset attributes.
Why can’t an external LLM API meet the reproducibility bar for regulated extraction?
Two reasons. Hosted frontier models are not deterministic: the same prompt returns different outputs across runs, and providers do not guarantee identical results even at temperature zero. And they are not pinnable: providers update models behind stable endpoints and retire versions on cycles measured in months, so an extraction run today cannot be reproduced in two years when the version no longer exists. Point-in-time reconstruction requires content-addressed model artifacts running on infrastructure you control.
What is the difference between the Bronze, Silver, and Gold tiers?
Bronze stores every ingested document and message immutably and losslessly, exactly as it arrived. Silver holds the clinical facts extracted from Bronze, each tagged with source coordinates, model version, and confidence. Gold holds the reconciled and deduplicated result, mapped to OMOP or FHIR, with every conflict resolution decision logged. Each tier can be re-derived from the one below it, which is what makes any value in Gold traceable to raw bytes.
How long does it take to build a fact-level provenance platform in-house?
With a senior team that has built data platforms before but not a healthcare-specific one, the realistic estimate is two to three years to land all 42 capabilities in a way that holds up under audit. The individual components are not exotic; the difficulty is that they interact, and several, including parallel identified and de-identified datasets, per-fact source coordinates, and point-in-time reconstruction, only work if they were designed in from the first sprint.
What is the test for genuine point-in-time reproducibility?
Given a query and a timestamp, the system should return the result it would have returned at that time, deterministically and in seconds, using the dataset versions, model and prompt versions, terminology releases, and business rules in force at that timestamp. If the answer requires reconstructing a snapshot or re-running a pipeline with archived parameters, the versioning is aspirational rather than real.