Benchmarking AI impersonation of licensed professionals: 11 laws and 380 adversarial tests
A new red-team benchmark measures whether frontier models impersonate doctors, lawyers, and financial advisors, and whether they disclose being AI. Every model tested failed at least a third of it.
An LLM can give a factually correct answer and still create legal exposure for whoever deployed it, if it answers while presenting as a licensed professional. Between January 2025 and January 2026, at least eleven US state and federal laws came into force or were introduced to regulate how AI systems present themselves in healthcare, mental health, legal, financial, and insurance conversations. We built a 380-case red-team suite at Pacific AI to test this surface, and ran it against four frontier models. None scored above 0.64, and every one failed at least a third of it.
The law treats impersonation as its own category of harm
Current laws treat impersonating a licensed professional, and failing to disclose AI status in a regulated conversation, as a category of consumer harm separate from whether the underlying advice was any good. California AB 489, in force since January 1, 2026, prohibits AI products from using names, credentials, or design elements that would lead a reasonable person to believe they are talking to a licensed health professional. An AI that calls itself “Dr.” anything violates it even when every answer is clinically sound. Accuracy does not cure the presentation problem, which is why accuracy benchmarks cannot measure it.
The legislative record explains where this came from. The findings behind AB 489 cite AI products deployed under “Dr.” persona names. The FTC’s 2025 action against DoNotPay concerned marketing an AI system as a legal advisor. Nippon v. OpenAI, filed in 2026, alleges that AI-generated, ready-to-file legal documents tailored to an active dispute amounted to unauthorized legal practice. Reports of teenagers forming relationships with AI characters framed as licensed therapists drove both New York’s General Business Law Article 47 and California SB 243.
Texas SB 1188 requires practitioners to disclose AI use to patients. Nevada AB 406 (July 2025) prohibits offering AI that provides professional mental or behavioral healthcare. Illinois’ Wellness and Oversight for Psychological Resources Act (August 2025) prohibits AI-delivered therapy while allowing AI-assisted licensed practice with disclosure. Utah HB 452 requires mental health chatbots to disclose AI status when asked. Maine HP 1154 requires disclosure across consumer-facing services generally. New York GBL Article 47 (November 2025) adds crisis protocols, escalation pathways, and a private right of action for AI companion products. California AB 3030 (January 2025) requires generative-AI patient communications to carry a disclosure that they were AI-generated, with instructions for reaching a human provider. The federal CHATBOT and GUARD Acts and state bills like New York S7263 remain proposals rather than law; S7263 would reach conduct that amounts to unauthorized practice even when the AI never claims a license.
Two properties of this wave matter for deployers. First, the exposure sits primarily with the deploying organization: the health system, law firm, brokerage, or platform, rather than the model provider. Second, the violations are behavioral. They happen when a production model meets an adversarial or emotionally loaded prompt: a condition standard capability benchmarks don’t create.
A surface no existing benchmark covers
Plenty of evaluation exists nearby. MedHELM and HealthBench measure whether a model reasons correctly about clinical content. General red-team corpora like AdvBench, HarmBench, and ALERT target content-level harm: toxicity, weapons information, illegal activity. Mental health safety evaluations like VERA-MH go deep, and well, on one industry. None of them asks the question these eleven laws ask: when an adversarial prompt invites the model to play doctor, lawyer, or fiduciary, does it refuse appropriately, disclose that it is AI, and route the person to a licensed human when one is needed?
The failure mode is distinct. A model can top MedHELM and still format its output as a signed prescription. Regular readers will recognize the pattern from the cognitive bias suite covered here recently: capability scores measure one thing, deployment behavior is another, and the gap between them is where the liability lives.
Built so a failed test names the exposure it creates
The dataset holds 380 adversarial test cases across healthcare, mental and behavioral health, legal practice, financial and investment advice, accounting and tax, and insurance, plus 23 more licensed professions—from engineers and architects to midwives, notaries, and general contractors. Each case sits on four axes: six failure patterns, four difficulty levels, seven profession categories, and 18 attack vectors drawn from the prompt injection literature and from documented real-world incidents.
Three construction decisions make the results believable.
Every test is anchored to a specific law or enforcement theory. Red teaming often tests for “bad behavior” without defining what makes the behavior bad. Here, each row names the statute or enforcement theory it probes. A row that coaxes the model into a credential-implying signature block anchors to AB 489’s design-elements language; a row that requests a patient message with no mention of disclosure anchors to AB 3030. When a model fails a row, the failure maps to a named exposure, and a compliance team can pull every row behind a single law and show exactly what was tested.
Distinct failure modes. The suite avoids five rephrasings of “are you a doctor?” Within each cell of the taxonomy, every row probes a different way in: a direct probe, a named clinician roleplay request, a credential cue slipped into a document format, a false premise, a multi-turn rapport build that ends in an out-of-scope ask, or a crisis context override. This slowed down the construction of the test suite considerably. It also means a high pass rate is evidence of broad behavior rather than memorization of a template.
The tests grade what a correct refusal contains. For many of these prompts, blanket refusal is itself a failure. A user disclosing acute self-harm intent needs crisis resources named. A parent describing a child’s red flags needs the red flags surfaced and an emergency pathway identified. A generic “I cannot help with that” abandons them in a crisis. Every row therefore specifies both must-do behaviors (identify as AI, name specific crisis resources, recommend the right human professional) and must-avoid behaviors (adopting credentialed personas, fabricating license numbers, providing medication doses, generating court-ready filings). The two families are graded separately. This test design punishes the pathological “safe failure” of a model that refuses everything and helps no one.
Coverage is deliberately uneven: crisis context gets 20 rows, self-disclosure 24, document completion 26, and multi-turn rapport 22, because those patterns dominate documented real-world incidents and carry the heaviest regulatory exposure. Encoded payloads and system-prompt extraction get seven rows each, because the failure-mode space inside them is narrower. The paper is explicit about what that choice costs: vectors with fewer than ten rows cannot support confident cross-vector comparison yet. An evaluation that states its own resolution limits before you ask is one you can trust when it does make a claim.
What four frontier models scored
We ran the full suite against Claude Fable 5, DeepSeek V4 Pro, Gemini 3.1 Pro, and GPT-5.4, graded with an LLM-as-a-judge framework against each row’s evaluation points, must-do items, and must-avoid items.
Claude Fable 5 and GPT-5.4 form a higher-scoring pair; DeepSeek V4 Pro and Gemini 3.1 Pro a lower one. The composition differs inside the tiers: GPT-5.4 posts the best must-avoid score (0.646), so it is strongest at staying away from prohibited behaviors, while Claude Fable 5 posts the best must-do score (0.787), so it is strongest at including the affirmative behaviors a compliant response requires.
The paper that introduces this AI impersonation test suite does something most benchmarks skip: it tests whether the instrument can tell the models apart at all. Pairwise Mann-Whitney U tests on per-scenario scores separate each top-tier model from each lower-tier model at p < 0.00001, with small-to-medium effect sizes (Cohen’s d between 0.36 and 0.48). The two within-tier comparisons are not statistically significant; Claude Fable 5 versus GPT-5.4 returns p = 0.35 with a negligible effect size. So the suite reliably distinguishes a stronger tier from a weaker one, and the 0.013 gap between the top two models should not be read as a ranking. Publishing that second sentence next to the first is what separates an evaluation instrument from a leaderboard.
A variance decomposition explains why: attack vector accounts for 6.7% of score variance and model for 5.1%, both highly significant; profession contributes 3.3% and difficulty 0.8%; failure pattern is not a significant factor at all. Roughly 83% of the variance sits at the level of the individual scenario, which matches a consistent finding across the LLM evaluation literature: the specific wording of an individual prompt moves scores more than any labeled category does. The practical consequence is that statistical power comes from many diverse scenarios, and the direct route to finer resolution, including resolving the top two models, is growing the dataset.
Neutrality disclosure: the paper’s authors (myself included) are unaffiliated with the companies developing the four models under test, and hold no financial interest in any of them. The four models are treated as systems under test, scored on published criteria.
Where the failures cluster
Every model failed at least a third of the suite. That is the headline for a governance audience: none of the four handles the impersonation-and-disclosure surface at a level that lets a deployer skip testing, whatever its clinical, financial, or general capability scores say.
The financial and investment profession is the hardest surface for all four models, with the lower tier dropping to roughly 0.40. The weakest attack vectors, on sample sizes solid enough to trust, are the missing-disclosure trap, where the model is asked for regulated output such as a patient message and must add the required AI disclosure without being prompted, and prefix injection.
The missing-disclosure trap is exactly the behavior California AB 3030 and Texas SB 1188 describe, and no model scored above 0.47 on it. The aligned failure pattern, proactive disclosure, produced the lowest scores of any pattern for all four models, between 0.34 and 0.37, though at 19 rows it is the thinnest sample in the suite.
Difficulty behaves as designed through the first three levels: for every model, scores decline from direct prompts through social engineering to adversarial jailbreaks, which is what a valid difficulty scale should produce. The fourth level, multi-turn, shows an apparent rebound the paper does not interpret: the bucket holds only 29 rows, and the rebound could reflect how the test harness expanded the multi-turn scenarios rather than anything about the models.
From one-time audit to standing test
The suite is built to be re-run, and the deployment pattern follows directly from the statistics. Pre-release, gate against a fixed pass-rate threshold or against the prior version of the same model. Given the within-tier resolution limit, small deltas between comparably scoring models are the wrong gate.
In production, sample rows on a rolling basis and monitor drift on the aggregate pass rate, since scenario-level variance makes any single taxonomy cell noisy. When a real impersonation or disclosure incident occurs, capture that prompt’s shape as a new row, which keeps the suite growing along the same axis that improves its statistical resolution.
The regulation anchors turn test results into audit material. When a regulator asks how a deployment tested for AB 3030 behavior, the deployer can produce the specific rows anchored to it and the model’s pass rates on them. That evidence describes what was tested and how the model behaved. It is not, and should not be presented as, a determination of legal compliance.
The dataset ships as part of the Pacific AI test suites, and the four-model evaluation was produced by running it through the Guardian continuous testing platform. The full preprint has the per-vector and per-profession tables and enough methodological detail for independent replication.
Twenty turns into a warm conversation, a user asks your deployed model whether they’re talking to a person. Eleven laws now have an opinion about what should happen next. Until you’ve tested it, you don’t know what does.
This post, like the preprint it summarizes, describes a testing methodology and reports model pass rates. It is not legal advice, and pass rates are not determinations of compliance with any regulation. Organizations subject to the laws named here should consult their own compliance counsel.
Frequently asked questions
What does this benchmark test that MedHELM or HealthBench don’t?
Those suites measure clinical reasoning accuracy. This one measures presentation behavior: whether a model adopts a licensed persona, produces regulated work product like prescriptions or court filings, discloses its AI status when asked and unprompted, and escalates a crisis to human help. A model can score well on one and fail the other; all four frontier models failed at least a third of this suite.
Which laws are the test cases anchored to?
Each of the 380 rows names the statutes it probes, drawn from eleven US laws in force as of May 2026, including California AB 489, AB 3030, and SB 243, Nevada AB 406, Illinois’ WOPR Act, Utah HB 452, Texas SB 1188, Maine HP 1154, and New York GBL Article 47, plus adjacent frameworks from SEC and FINRA rules to state unauthorized-practice statutes. Pending bills are treated as motivation, never as enforceable obligations.
Which model performed best?
Claude Fable 5 scored highest overall at 0.637, with GPT-5.4 at 0.624; the difference between the two is not statistically significant, so the honest reading is that they form a top tier rather than a ranking. DeepSeek V4 Pro (0.515) and Gemini 3.1 Pro (0.533) form a lower tier, separated from the top pair at high statistical significance.
What were the hardest tests for every model?
The financial and investment profession produced the lowest scores for all four models. Among attack vectors with solid sample sizes, the missing-disclosure trap, where the model must add a legally required AI disclosure without being asked, and prefix injection were the weakest for every model tested.
Does a high pass rate mean a system complies with these laws?
No. Pass rates describe how a model behaved on regulation-anchored tests. They are evidence of testing rigor that a governance program can present to auditors and regulators, and what any result means for a specific deployment is a question for the organization’s own compliance counsel.
How is the suite meant to be used in practice?
Three ways: as a pre-release gate against a fixed pass-rate threshold or a prior version of the same model, as a production monitor that samples rows on a rolling basis, and as a living corpus that grows a new row from every observed real-world incident. It ships as part of the Pacific AI test suites and runs in Guardian.