Medical records could be revealed by AI training-data vulnerability

**Benjamin Thompson** (0:24)
Welcome back to the Nature Podcast. This week, how medical information could be stolen from AI training data.

**Nick Petrichow** (0:31)
And the strongest evidence yet that the universe is uneven. I'm Nick Petruchow.

**Benjamin Thompson** (0:38)
And I'm Benjamin Thompson.
First up on the podcast this week, we've got a story about cyber attacks on medical AI systems, attacks that could allow criminals to steal someone's personal information from the AI's underlying data sets. Now, AI systems are increasingly being developed, tested and used in medical contexts. For example, to quickly assess whether a skin lesion might be cancerous and needs to be checked by a clinician.
These systems could be invaluable for aiding in diagnoses and helping patients access quicker care when needed. But these systems don't start off knowing what to look for. They need to learn from pre-existing images and information. And as Moritz Knoller from the Technical University of Munich in Germany explains, they're only as good as the data they're trained on.

**Moritz Knoller** (1:39)
I think what has been shown to be the driving force for the success of these systems is really the availability of suitable data and very, very big amounts of it. What people find is that as you increase the amount of data and the amount of computation, these systems tend to get better and better and better.

**Benjamin Thompson** (1:54)
Specifically, these AI models are often trained on publicly available data sets of medical information that patients and health services have allowed to be used for research. These data are used to train the AI to form patterns and connections, but the use of this data can come with risks.

**Moritz Knoller** (2:13)
As we move towards these systems that are trained on bigger and bigger data sets, what people are worried about is that when you make these models available, that to some degree, you might also expose the data that they were trained on.

**Benjamin Thompson** (2:27)
Moritz is part of a team who have a paper in Nature this week examining something called a Membership Influence Attack, or an MIA.
MIAs are ways of probing the data used to train an AI. Moritz and the team wanted to see if feeding a piece of someone's medical information into an AI would reveal whether they were part of the data set used to train it. On the face of it, this might not seem like a big problem. Well, let's say a criminal already has some, but not all of your medical information, and then puts it into an AI model to find out if this information has been included in the training data. It turns out that from this, they could learn a lot.

**Moritz Knoller** (3:06)
We made this example with a model that's trained basically to predict based on some blood data, so this general blood results, would this patient respond positively to immunotherapy or chemotherapy. And if I know that you were part of a training data set of chemotherapy patients, I now know that you have cancer. Something that I couldn't otherwise learn from your blood data alone.
So I learned something new about you, and this is sensitive information potentially.

**Benjamin Thompson** (3:37)
Moritz wanted to work out how much of an issue MIAs might be to individuals in this kind of context. So he and his colleagues conducted their own attacks on a series of medical AI models. These had been trained using seven data sets of publicly available anonymized medical data. Things like chest x-rays, skin images and mammograms.
Different models were trained using different subsets of information. The team then fed the AIs a particular piece of data and tried to work out whether it was included in the training set or not.

**Moritz Knoller** (4:09)
So what you look at is essentially just how the predicted probability changes based on whether a data point was included in a model's training data set or not. And how you do that is you simply query the model. So you send a request to the model.
Say, let's say, with your chest x-ray and you look at the prediction and check, is this prediction different when you're a part of the data set or not? And you look at how does your predicted probability of having pneumonia change whether you were in a data set or not?

**Benjamin Thompson** (4:41)
It turns out that AIs are much more confident when making predictions about data they've seen before.
Essentially, if you gave it a chest x-ray of someone with pneumonia and asked, does this person have pneumonia, you'd get a higher percentage chance if the AI had been trained using this image. And by using statistical methods, the team were often able to identify if a person had been included.