–Little agreement found when 12 wearable devices were compared to gold standards
Fitness trackers have not been found to help people improve the way they exercise, an article in the Sunday New York Times noted in a summary of the case against wearable fitness trackers, pointing up technical and even philosophical limitations of the devices. Now a new paper published in JAMA Internal Medicine shows that these devices are not even correctly performing their most basic function, which is accurately measuring physical activity and estimating energy expenditure.
Researchers in Japan compared 12 wearable fitness devices, including the popular Fitbit and Jawbone devices, to two “gold standard” research techniques, the metabolic chamber method, which captured all metabolic activity in a 24-hour period, and the doubly-labeled water (DLW) method, which captured metabolic activity in a real-life setting over 15 days. The 19 healthy adults participating wore all 12 devices while undergoing the two gold standard methods. The devices and methods were then used to calculate energy expenditure.
The measurements of a day’s total energy expenditure (TEE) differed widely from device to device. TEE was 2,093 calories as measured by the metabolic chamber and 2,314 calories as measured by the DLW method. (The senior author of the paper, Motohiko Miyachi, PhD, of the National Institute of Health and Nutrition in Tokyo, explained to me why the two gold standard methods differ: DLW, because it is able to track energy expenditure through the course of normal daily life, detects the effect of more activities than can be tracked in the structured-living setting of a metabolic chamber.)
Compared to the metabolic chamber, the TEE measured by the devices ranged from 278 calories lower to 204 calories higher per day. Compared with the DLW method, all the devices underestimated energy expenditure, from 590 calories lower to 69 calories lower per day. The researchers speculated that the consistent underestimation may have been because the subjects did not wear the devices while bathing or during battery charges.
They concluded that their findings “suggest that most wearable devices do not produce a valid measure of total energy expenditure.”
A Series Of Disappointing Events
The findings might shock and disappoint the large number of people who have bought these devices. In 2015, according to the Times, the market for wearables doubled from the year before to reach $1.5 billion. Many think this is just the start, with at least one analyst predicting an astonishing $50 billion in sales in just a few years.
But the results of the study almost certainly won’t shock researchers in the field who have spent a lot of time trying to measure exercise and energy expenditure. Michael Joyner, MD, of the Mayo Clinic in Rochester, Minn., noted that these experts have expressed “concern for several decades or more about wearables that rely primarily on accelerometers,” which is the underlying technology behind the mobile fitness trackers used in the study. Even devices that have been “validated” for use in research were not particularly reliable in the study.
Joyner further noted that the differences between the devices is also “troubling,” and that this problem will likely “get worse when more heterogeneous groups are assessed.” In other words, as devices are used by more and more people belonging to populations for whom the devices have not been designed, the results will become even more unreliable.
The new study is only the latest to turn up disappointing results for mobile health devices and apps. Earlier this month a paper in JAMA Internal Medicine showed that a best-selling blood pressure app for mobile phones delivered completely unreliable results. Also recently, the first prospective randomized trial in the field, led by Eric Topol, MD, of the Scripps Translational Science Institute in La Jolla, Calif., found no health benefits in people using smartphones to monitor blood pressure, diabetes, and arrhythmias.
“There is little evidence that extreme monitoring improves outcomes” or that “these devices are going to motivate long term behavior change,” said Joyner. “In many cases, they are high-tech versions of old-school exercise equipment from ‘last Christmas’ that ends up unused in the garage or basement.” In the New York Times story on mobile tracking devices Joyner suggested a far less expensive and lower technology alternative with proven benefits: the jump rope.
The Data-Driven Patient
“Measuring energy expenditure is very challenging, as is shown by the different results from the gold standards here,” said Timothy Plante, MD, of Johns Hopkins. He said that these devices might be useful in a small subset of patients “who are driven by data trends. These are the patients that come in with graphs of their blood pressure trends over time, which they have dutifully been logging in a spreadsheet for months and months. I think that giving these folks a tool to follow their energy expenditure, or an estimation of their overall activity, can help them increase their average activity because they’ll start noticing trends where they are low. For example, if they notice that they are burning fewer calories on the weekend, they may choose to go for a walk or add in more time at the gym. I always advise them that these devices are far from perfect, as this study by Dr. Murakami and colleagues clearly shows. As long as the devices are consistently biased in one direction, but increase or decrease relative to themselves, I think they are still useful for these patients.”
Plante added one important caution: “Patients can get into trouble if they are misinformed about the amount of calories they have burned and use that to treat themselves to foods they otherwise may not eat. If a device tells a person that they are burning a few thousand more calories in a week than they actually did, they may opt to go for the big desert or have a hamburger rather than sticking with a healthier option. Over time, this can lead to bad dietary trends and weight gain or failure to lose weight despite best efforts. For anyone using these devices, you need to take the results with a Morton canister of salt.”