Your phone can use tiny skin-colour changes to measure your heart rate

**Benjamin Thompson** (0:24)
Welcome back to the Nature Podcast. This week, how your smartphone's camera could measure your heart rate, and testing Richard Feynman's solution to the restaurant dilemma problem. I'm Benjamin Thompson.
First up on the show this week, reporter Julie Gould has been finding out about a way that a person's mobile phone could be used to quickly and unobtrusively measure their heart rate.

**Julie Gould** (0:57)
Resting heart rate, literally the number of times someone's heart beats in a minute while at rest, is a pretty simple number to calculate. But it carries a lot of information about a person's cardiovascular health. A declining resting heart rate over time can signal improving fitness, but a consistently elevated resting heart rate can be associated with cardiovascular disease, diabetes and even early mortality.
So knowing if this rate is increasing can be useful.

**Ming Zhe Po** (1:25)
Having the ability to be notified if you see trends are different, your heart rate, resting heart rate is different. It has been increasing over the last couple of months. That could trigger some further investigations as to why. Is it stress build up or is it something else going on in your health?

**Julie Gould** (1:43)
This is Ming Zhe Po, a staff research scientist at Google. His goal is to simplify taking these measurements so that these warning signals can be picked up faster and, if need be, treated sooner.
But clinical measurements of these numbers can be time consuming. It requires dedicated time and effort to go to the doctor or to take a few minutes out of your day to rest, measure and make a note of it. There are devices already out there that make it easier, like smartwatches which track resting heart rate without any effort from the user. But these devices cost money, require charging and need to be worn consistently. A large portion of the world's population, particularly in lower income countries, simply doesn't have access to them.
But what if something that is ubiquitous could do the job? Well, this week Ming is part of a team who have a paper out in Nature reporting a system that estimates a person's resting heart rate simply from brief videos captured while they're using their phone. It's called PHRM or Passive Heart Rate Monitoring, and it uses machine learning to measure resting heart rate during everyday smartphone interactions without the need for using a specific app.

**Ming Zhe Po** (2:50)
PHRM is designed to run entirely on your smartphone device.
Every time you unlock your phone, to use it, to check the email or to browse the web, it activates the front-facing camera to capture a very short video clip, about eight seconds. We train a deep neural network to analyse the video frames captured in this short video clip to infer your heart rate at that moment. And at the end of the day, we can aggregate all these intermittent readings throughout the day to estimate your resting heart rate.

**Julie Gould** (3:23)
The technology being used by the front-facing camera is called remote photoplethysmography.

**Ming Zhe Po** (3:28)
Every time your heart beats, it generates this pulse wave that propagates through the blood vessels and changes the blood volume, including the vessels that are underneath the skin of your face.
So blood absorbs specific wavelengths of light, primarily around the green-yellow spectrum. So these are the subtle color changes that the front-facing camera, the smartphone camera is analyzing to estimate your heart rate. Human eyes can't really pick up on these subtle color shifts, because the changes are on the order of something like 1% of change of the baseline. But a digital sensor is able to, by measuring the underlying raw pixel intensity and tracking the changes over time.

**Julie Gould** (4:09)
Ming and his team then trained a neural network to estimate the heart rate from the videos using these minute color changes brought about by pumping blood. But real-world smartphone use introduces huge challenges. People move, they walk, or they sit in cars, and lighting is constantly changing.
So to address these challenges, the team built in automatic image stabilisation and cropping to ensure a person's face was the focus. And they also allowed the system to assess how good it thought each video was.

**Ming Zhe Po** (4:39)
The way we train the deep neural network is to instead of just allowing the network to output a single heart rate number, we also allowed the network to output a confidence estimate.
So in other words, we allowed the network to express uncertainty. So if conditions were really bad and inconducive for a reliable heart rate measurement, the output confidence would be low and the algorithm could then discard that unreliable measurement. We conducted a real world study with participants using their personal phones in unconstrained conditions. And these two aspects are important, I'd say. Both the personal phone use and unconstrained use.