Over the past two months my team benchmarked the new general-purpose privacy tools on clinical text: OpenAI’s Privacy Filter at 0.55 PHI F1, Microsoft Presidio at 0.60 to 0.85 F1 in independent peer-reviewed studies, Databricks’ `ai_mask()` at 0.71, and the frontier LLM APIs at 0.86 to 0.91, against 0.96 for a healthcare-specific pipeline on the same corpus, which also holds 0.98 recall on a second 381,959-token clinical corpus and 0.98 micro F1 on the official 2014 i2b2 test set. The per-entity tables are on the John Snow Labs blog. This post is about how to read them, because de-identification is one of the few tasks in machine learning where the reading is pass or fail. A summarizer at 0.85 saves a team real effort. A de-identification pass at 0.89 produces a corpus that no privacy officer can release, which is the same thing a pass at 0.55 produces.
The legal standard: no PHI or very low risk of it
The regulation does not ask for a score. HIPAA offers two paths to de-identified status: Safe Harbor, which requires that all 18 identifier categories be removed, and Expert Determination, in which a qualified statistician certifies that the risk of re-identification is very small. Under GDPR, data that has been pseudonymized rather than anonymized remains personal data inside the full regulatory perimeter. Both frameworks are binary. No tier of either recognizes 80% removal and grants 80% of the permissions, and there is no version of “mostly de-identified” that a downstream researcher, partner, or model-training pipeline is allowed to touch.
Neither framework names an accuracy number, because neither was written about software. The field calibrated instead against the only prior benchmark that existed, careful manual de-identification by domain experts, and Yogarajan, Pfahringer, and Mayo’s 2020 review identifies 95% F1 on the 2014 i2b2 corpus as the level widely treated as equivalent to it.
95% is hard to reach with people
That bar sits above what individual experts achieve. Neamatullah et al. measured 14 clinicians de-identifying nursing notes by hand: recall ran from 0.63 to 0.94 by clinician, a single annotator averaged 0.81, two-annotator consensus reached 0.94, and even a consensus of three experts failed to remove all PHI. Building the i2b2 gold standard, Stubbs and Uzuner found individual annotators averaged 0.93 token F1 against the consensus they collectively produced, and the corpus reached gold quality only through double annotation of every record, arbitration, and rounds of sanity checking and proof reading. A single reviewer is not a safety net: one identifier in five gets past them.
What a recall number means at a million notes
Recall carries the regulatory risk and its implication multiplies across a corpus. A clinical note carries roughly 31 PHI mentions.
* Chunk-level detection, counting partial overlaps as found; on exact matches only, ai_mask() sits at 0.51.
The last column is recall^31, the chance a note comes out fully clean if each mention were independent, which overstates the damage since a repeated name is one detection problem rather than eight (read it as a worst case). The direction survives any correction. Below about 0.90, almost no document in a corpus is clean, and no downstream process recovers it. Above 0.95, the residue becomes small enough for adversarial testing and statistical risk analysis to characterize, which is what Expert Determination requires.
John Snow Labs’ pipeline sits above that line on every corpus tested: 0.98 recall in the table above, 0.98 micro F1 on the official i2b2 test set, and over 99% detection across 2 billion notes at Providence. Between 0.72 and 0.98 there is no plateau at which the data becomes shareable.
0.89 fails the same way 0.55 fails
The frontier models did improve. GPT-4o scored 0.79 PHI F1 in our Text2Story 2025 evaluation; GPT-5.5 scores 0.89 on the same corpus today. Read as a trend line, that number invites a plan: wait a version or two and let the general models solve it. Read against the bar, it says something different. 0.79 was below the threshold and 0.89 is below the threshold, and both produce a corpus that holds no de-identified status and cannot leave the security perimeter. In this task, improvement below the bar has a regulatory value of zero.
The residual errors also sit in the worst possible place. The aggregate hides it and the per-entity tables show it: Claude Opus 4.8 scores 0.49 on contact identifiers and 0.68 on age, and Gemini 3.1 Pro scores 0.60 on age, against 0.95 and 0.97 for the healthcare pipeline. Contact and age are enumerated Safe Harbor categories, and age carries its own aggregation rule above 89. These identifiers are defined by their role in a document rather than their surface form: phone and fax numbers live in headers, footers, and referral blocks, and record numbers appear as MRN, MR#, and a dozen local conventions. That’s a structural mismatch with general training distributions – and why the last points of the gap are the hardest ones, concentrated on the categories the regulation defines.
In addition, a model that cleared the number would still not clear the bar, because the bar includes validation. Expert Determination is built from span-level detection records, confidence thresholds, versioned models, and reproducible runs. A frontier API that returns text, may be swapped or retired by its vendor, and cannot be pinned to the version that processed a 2026 cohort resets that evidence with every release.
Beyond accuracy
There is a second independent reason a strong score settles nothing, which is why an earlier paper of ours on production de-identification is titled “Beyond Accuracy”. Every tool in these benchmarks can find PHI and delete it. A de-identification program needs eight other things.
Suppose ai_mask(), at 0.71 today, scored 0.96 tomorrow. It would still fail nearly every de-identification program I have seen, because masking is its entire feature set. It returns no spans, so there is no record of what was detected, no confidence to threshold on, and no way to sample detections for review. Those artifacts are what an Expert Determination analysis is built from. It also maintains no consistency, so a patient masked in two notes cannot be recognized as one patient. The same reasoning applies in some degree to every general-purpose tool in the table. The benchmark scores describe the first row. A de-identified medical dataset requires more.
Masking is just where de-identification starts
Replacing every identifier with [MASKED] produces a compliant corpus that has lost most of its research value. A longitudinal cohort needs the same patient recognizable across a decade of notes, claims, PDFs, and DICOM headers, intervals between events preserved so a time-to-progression analysis still means something, and a narrative readable enough to extract from, which “patient was seen by [MASKED] at [MASKED] on [MASKED]” is not. Three capabilities do that work.
Consistent obfuscation. PHI is replaced with realistic, gender- and context-aware surrogates, deterministically. If “Jane Sunshine” becomes “Anne Boleyn,” every later “Jane” maps to “Anne” across the corpus.
Deterministic tokenization. An MRN, or a name-plus-birth-date composite, becomes a cryptographic hash, so records about the same person written years apart link without exposing the identifier. Without this, no analytics that track a patient over time can be delivered.
Date Shifting. Generalizing or deleting dates prevents calculating even simple analytics like 30-day readmissions, or answering simple questions like what happened before what. Shifting dates on a random number of days, that is consistent across all records of the same patient, is essential.
Multimodal linkage. The same tokenization and obfuscation applied across EHR text, claims, PDF reports, DICOM images, and structured tables, with date shifting that moves dates while preserving intervals.
All of it must be done together. Jane stays female because she has a history of breast cancer, and “April 2020” shifted by a random number of days comes back as “March 2020” rather than “3/3/2020.” Imaging raises the bar again, because PHI in a DICOM file lives in the metadata headers and burned into the pixels simultaneously, and handling only one leaves the study identified. A dual-level approach we published at CVC 2026 reports 100% success on tag retention, date shifting, and UID consistency on the MIDI-B dataset, with text processing accuracy above 99.9%. None of that work appears in any text benchmark table.
At Providence, this full stack ran across 2 billion patient notes at over 99% PHI detection, validated through three months of external red-teaming, manual review of 35,000+ notes, equity analysis across gender, age, ethnicity, and geography, and adversarial re-identification testing on 790 patients that produced zero re-identifications. The methodology is published. Detection accuracy is what gets a program to the starting line; consistency validation, red-teaming, and statistical risk analysis are the work that follows.
Two questions to ask your de-identification vendor
First, ask for per-entity recall on your own notes. Every tool above has an aggregate that sounds respectable and a category where it finds fewer than half the identifiers: ai_mask() finds 35% of contact identifiers behind a 0.71 headline, and Privacy Filter finds one address in four. Second, ask about everything after detection: whether it can obfuscate consistently across a corpus, tokenize deterministically for linkage, shift dates while preserving intervals, return spans for audit, and follow the same patient across modalities.
De-identification is one of the few tasks in healthcare AI where the requirement comes from law rather than from a stakeholder conversation. Below the bar, every score is the same score, because the output has the same status: identified. There is no partial credit, and no version of this problem in which 0.89 is a good start.
Frequently asked questions
The frontier models are improving fast. Should we wait for the next version?
Waiting is a bet that a general model will master identifiers defined by clinical context rather than surface form, which is where the remaining errors concentrate: contact identifiers at 0.49 to 0.69 and age at 0.60 to 0.93 for the current frontier models. It is also a bet on the wrong variable, because clearing the accuracy number does not produce de-identified data. That requires all 18 Safe Harbor categories covered, span-level detection records, consistent tokenization and obfuscation, and validation against a pinned model version – none of which an unpinned API provides.
If a model reaches 0.96 F1, is de-identification solved?
Detection at that level clears the accuracy threshold, and a program still needs surrogate obfuscation, cross-document consistency, deterministic tokenization, date shifting, span-level audit records, and validation evidence for Expert Determination. Those are separate capabilities from detection.
Isn’t masking enough if I only need a compliant corpus?
Only if nobody needs to analyze it afterwards. Masking yields a corpus in which every patient is unlinkable, every date is gone rather than shifted, and every narrative has holes where the context used to be. Longitudinal research, cohort building, and time-to-event analysis all need consistent obfuscation and deterministic tokenization instead, and most general-purpose tools have neither.
A vendor’s masking service is HIPAA compliant. Does that make the output de-identified?
No, the two claims describe different things. Service-level HIPAA compliance means the processor will sign a BAA and handle PHI appropriately. Whether the output is de-identified depends on detection recall, coverage of all 18 Safe Harbor categories, and the ability to document what was detected, none of which follows from the service claim.
Does masking satisfy GDPR?
Masking produces pseudonymized data, which remains personal data under GDPR and stays inside the full regulatory perimeter. Anonymization requires that re-identification be reasonably impossible, a higher bar that depends on the whole pipeline and the release context rather than a single model, and it is a question for your own counsel rather than a vendor.
Where are the detailed benchmark tables?
On the John Snow Labs blog: the Privacy Filter comparison, the Presidio comparison, the cloud API and LLM comparison, and the de-identification solution page, with per-entity precision, recall, and F1 for every system discussed here.