The cart is driving the horse: Misguided performance measures

When it comes to measuring quality, there is a huge disconnect between payers and physicians. Public and commercial payers are putting an increasing emphasis on value, as defined by quality performance measures. However, the majority of physicians don’t believe that these measures accurately represent their quality of care. 

A recent analysis in the New England Journal of Medicine lends support to the physicians’ side of the issue. Three researchers and members of the American College of Physicians (ACP) Performance Measurement Committee used a newly developed set of criteria to evaluate measures included in Medicare’s quality payment program.

The review criteria include the validity of evidence the measurement is based on, whether the measurement and exclusions are clearly specified, the feasibility of the measurement, and whether implementation of the measure will have a meaningful impact on clinical outcomes. The ACP criteria also evaluate whether the quality measure will promote appropriate care by reducing an overused service or encouraging an underused one. 

Although it seems obvious that a quality measurement should be clearly specified, feasible, and have a meaningful positive impact, unfortunately the majority of quality measures did not fulfill these basic criteria when scrutinized. Out of 86 measures of quality for general internal medicine, the researchers rated just 32 (37%) as valid. A little more than one third (35%) were rated as not valid, and the rest were rated as having uncertain validity.

Most of the measures that were ranked as not valid lacked evidence showing that the test or treatment actually improves outcomes. Many others that were ranked not valid or of uncertain validity were applied too broadly over the population, not adequately specifying who should or should not receive the intervention.  

How can it be that so many measurements aren’t meaningful or accurate? Part of the answer is that everyone is eager to improve quality, so more and more measures get added, but many are implemented before they are properly evaluated. 

Another reason is that certain outcomes or practices are easier to measure than others, so they are prioritized over harder-to-measure outcomes (although they don’t have as large an impact on quality). This is sometimes called the “Streetlight Effect” – looking for something where the light is better, not where the lost item is likeliest to be.  

Dr. Yul Egnes, internal medicine physician and former chair of the ACP, mourned this mistake in a KevinMD guest blog a few years ago:

“It was supposed to be that if we provided high-quality care to our patients, the measurements would reflect that. Instead, the mantra is that if we score well on our measures, then that means that we provided high-quality care. In other words, the cart has become the horse.” 

The authors of the NEJM study think it’s time to slow down the horse and cart. They recommend taking a “time-out during which to assess and revise our approach to physician performance measurement.” Not just for the sake of doctors struggling to check all the boxes, but for patients who have been and will be harmed by misguided measures.