The Vast Gap Between Medical Records and Medical Truth

This article addresses the limitations of clean medical data for training AI models, emphasizing gaps in patient records, self-medication, doctor-patient interactions, and the potential for fraud.

“When I grow up, I will never, ever lie.”

The child who said this to me has since grown out of diapers, developing an astonishing mind and appreciation of the rich complexity of life. This saying stayed with me as the most naïve thing I’ve ever heard — until last year when I read a broad email following a major data normalization project on medical claims and electronic medical record (EMR) data from two hospitals: 

“We produced a clinical dataset that is perfectly clean.”

The data was transformed, joined, and validated by a top-tier enterprise data integration tool. All values were now conforming to their defined schema, parsing correctly, stripped of extreme outliers, without gaps in time coverage, normalized to the same versions of the same code sets, and validated for the consistency of each field and relationships between fields.

It was a big effort and a big win by a data analysis team. It demonstrated domain expertise and a good understanding of the team’s historical customers: business intelligence (BI) teams who built dashboards, prepared reports, and answered ad-hoc queries.

It also made my list of the most naïve statements of all time. It showed how inadequate these tools and techniques are concerning preparing data for training predictive models. BI reports that are viewed by human experts need to correctly query the data from a database. AI models need to know what happened to a patient in order to give a correct recommendation. There’s a world of difference.

Medical records don’t reflect people’s full health history

If an EMR record states that a patient has these medical problems, is taking certain medications, and has certain lab results, it’s unlikely that these resources capture the full picture. Here’s how this happens:

1. People don’t always go to their doctors when they get sick. According to a 2018 study, they want to save money (44%), think that nothing is wrong with them (36%), decide to wait it out (31%), don’t want to take time off work (27%) and have other excuses they choose from.

2. When people are sick, self-medication is sometimes their first option. U.S. consumers make 26 trips a year to purchase over-the-counter (OTC) products but only visit doctors three times a year, while many prefer natural remedies.

3. When people go to the doctor, they often don’t discuss everything happening with them. They go with a goal — to get a referral, get a prescription, etc. Doctors also don’t always have the time: Appointments average from 48 seconds in Bangladesh to 22.5 minutes in Sweden and 20 minutes in the U.S.

4. Doctors do not always document everything. They focus on what the patient needs to get their prescriptions or lab orders. Healthcare record integrity is a major issue — covering templates, copy/paste, dictation errors, amendments, and more.

5. Most people lie to their doctors. Up to 80% of Americans do, mostly because they’re embarrassed.

Medical records can be misleading

Most of the above still assume that what appears in the medical record is correct. It’s important to remember that’s often not the case:

1. Vital signs depend heavily on how they’re measured. This is especially critical when joining data from multiple hospitals. Blood pressure changes depending on whether it’s taken before or after meals. Temperature readings depend on the type of thermometer. And pain levels depend on the gender of the patient and the nurse.

2. Clinical codes — the core list of diagnoses and procedures — can vary by 50% or more when the same patient visits are given to different coders.

3. The prescribed drugs and lab orders are not indicators that the patients took them. Approximately 50% of Americans don’t take their medication as prescribed. Within a year, one in two diabetics stop taking their meds, and one in seven diabetics don’t complete lab orders within six months of getting a referral.

3. Even within a hospital, the meaning of timestamps — such as when medications were given — is not the same across wards and providers.

4. Outright fraud is estimated to be about 3% of U.S. healthcare spending. This means, for example, medical records that are falsified or records of events that never happened.

Designing for Low-Quality Data

It means you must be extremely careful about making inferences from medical data.

This isn’t a big problem for BI systems because the people who read the reports know what the reality is and correct it. They know if contagion, new thermometers, or a broken AC system caused a patient’s temperature spike. They know if better clinical quality metrics result from healthier patients or a documentation improvement project by the revenue cycle team. They know that patients lie, disagree, and change their minds.

There are solutions: E-commerce sites must deal with bot traffic. Online ads deal with fraudulent clicks, and social media sites deal with fake accounts. They’re not simple solutions, and they tackle hard problems, but they work in real systems every day. They’re far better than ignoring the problem or keeping a human in the loop since that human will lose trust in your system early on and learn to ignore it. How will your AI models deal with this reality of low-quality data?