When studies collide: Making sense of contradictory research findings

I know blogs are supposed to be current – otherwise, what’s the point of posting entries that get archived after a few weeks? However, every once in a while I come across a resource from a year or two back, which is so useful I feel the need to share it. Such is the case with this article from the New York Times Science Times. It shows how a journalist can do a superlative job of helping the public understand the complexities of science.

NYT Science Times published an invaluable special issue in 2008 entitled “Decoding Your Health.” The issue responded to the huge amount of medical information available now to consumers on the web, in the press, and in the doctor’s office. The articles are very helpful in “decoding” all this information, and deciding what is useful and what isn’t.

One particular article, however, really grabbed me: “Searching for Clarity: A Primer on Medical Studies.” I’ve rarely seen such a good job of laying out the kinds of studies we should trust, and how medical evidence accumulates to create guidelines for what people should do.

They take an example which could serve as the poster child for the dilemmas consumers face. In the 1990s, everyone was enthusiastic about the idea that the antioxidant beta carotene, which is found in certain fruits and vegetables (such as carrots, squash, apricots, and green peppers), could be good for your health. And this idea was backed up by some animal and observational studies suggesting that beta carotene protected against cancer. Supplement makers had a heyday selling beta carotene capsules.

Then it happened: results were published from three large, very well-done clinical trials, in which people were randomly assigned to take beta carotene or a placebo. These findings showed that beta carotene supplementation not only didn’t prevent disease, but it might even place people at greater risk of cancer.

If you were watching TV back then, you may remember seeing Frankie Avalon on a commercial (for you youngsters, Frankie was a 50’s teen idol with such hits as “Cupid,” “De-De-Dinah,” and “Tuxedo Junction”). As the article notes, he sat in front of a big pile of papers that said “beta carotene works,” and a tiny pile representing the three studies showing it doesn’t. The message: Who are you going to believe?

The answer is: the clinical trials. The article lays it out clearly, showing that there are three fundamental principles that make a more definitive study:

  • You have to compare like with like: “the groups you are comparing must be the same except for one factor — the one you are studying.”
  • The bigger the group studied, the more reliable its conclusions. They make a very helpful point: scientific studies don’t come up with a single number; instead, they come up with a margin of error (like you have a 10-20 percent reduction in risk). Larger numbers = greater certainty.
  • And the finding should be plausible. There should be some supporting evidence for the finding, such that it doesn’t come out of nowhere.

This is a good article to pass along when you are presenting scientific findings that contradict deeply-held beliefs. It shows that when it comes to research on health, more studies aren’t necessarily better – it’s having the right kinds of studies.

Speak Your Mind

Skip to toolbar