Randomized, controlled designs: The “gold standard” for knowing what works

You’re having trouble sleeping one night, so you finally give up and turn on the TV. It’s 2 AM, so instead of actual programs, much of what you get are informercials. As you flip through these slick “infotainment” shows, you hear enthusiastic claims about the effectiveness of diet pills, exercise equipment, and a multitude of other products

You will soon see that almost every commercial uses case studies and testimony of individuals for whom the product has supposedly worked. “I lost 50 pounds,” exults a woman who looks like a swimsuit model. “I got ripped abs in 30 days,” crows a man who, well, also looks like a swimsuit model.

The problem is that this kind of case study and individual testimony is essentially worthless in deciding if a product or program works. The main problem is that it’s very hard to disprove case study evidence. Look at the informercials – they seem to have worked for some people, but what about all the people who failed? And how do we know that the people who lost weight, for example, wouldn’t have done so without buying the product?

So case studies and testimonials aren’t worth much because they don’t give us the kind of comparative information needed to rule out alternative explanations.

To the rescue comes experiments using randomized, controlled designs (RCD). Such experiments are rightly called the “gold standard” for knowing whether a treatment will work. In a RCDs, we create a test so that one explanation necessarily disconfirms the other explanation. Think of it like a football game. Both teams can’t win, and one eventually beats the other. It’s the same with science: our knowledge can only progress if one explanation can knock out another explanation.

 The main weapon in our search for truth is control group designs.  Using control groups, we test a product or program (called the “treatment”) against a group that doesn’t get whatever the treatment is.

 Case studies simply don’t have the comparative information needed to prove that a particular treatment is better than another one, or better than just doing nothing. And that’s important because of the “placebo effect.” It turns out that people tend report that a treatment has helped them, whether or not there is any actual therapy delivered. In medicine, placebo effects very strong, and in some cases (like drugs for depression) the placebos have occasionally been found to work more effectively than the drugs.

 So what is a randomized, controlled design? There are four components of RCDs:

 1. There is a treatment to be studied like a program, a drug, or a medical procedure)

 2. There is a control condition. Sometimes, this is a group that doesn’t’ get any treatment at all. Often it is a group that gets some other kind of treatment, but of a different kind or smaller amount.

3.  Now here’s the key point:The participants must be randomly assigned to treatment or control groups. It is critical that nobody – not the researchers, not the people in the experiment – can participate in the decision about which group people fall into. Some kind of randomization procedure is used to put people into groups – flipping a coin, using a computer, or some other method. This is the only way we can make sure that the people who get the intervention will be similar to those who do not.

4. There must be carefully defined outcome measures, and they must be measured before and after the treatment occurs.

Lots of the bogus claims you see on TV and elsewhere look only at people who used the product. Without the control group, however, we can’t know if the participants would have gotten better with no treatment at all, or with some other treatment.

Catherine Greeno, in an excellent article on this topic, sums up why we need to do RCDs if we want to understand if something really does or doesn’t work. She puts it this way:

  • We study a treatment compared to a control group because people may get better on their own.
  • We randomly assign to avoid the problem of giving worse off people the new treatment because we think they need it more.
  • We measure before and after the treatment so that we have measured change with certainty, instead of relying on impressions or memories.

 So when you are wondering if a therapy, treatment, exercise program, product, etc. are likely to work, keep those three little words in mind: Randomized, Controlled Design!

The science of political campaigns

The 2012 presidential candidates will spend billions of dollars to encourage people to vote for them this year.  Independent analysts currently estimate that President Barack Obama and Republican nominee Mitt Romney will spend more than $6 billion in their campaigns. But how do they really know that money will bring them votes?

It turns out there is an emerging cadre of researchers using scientific techniques – specifically the randomized controlled trial – to find out which campaign strategies actually bring people to the polls.

Among them are Todd Rogers, a behavioral psychologist at Harvard University, and David Nickerson, a political science professor at the University of Notre Dame. Together, they conducted a study in the 2008 Democratic primary in Pennsylvania that attempted to determine whether get-out-the-vote reminder calls made a significant difference in voter turnout.

For the study, Rogers came up with some questions that would ask people about their plans on voting day such as “What do you think you’ll be doing before you head to the polls on Tuesday?” and “Where do you think you’ll be coming from that day?” His theory – based cognitive psychology principles – was that these questions would plant a seed voters’ minds to helps them remember to vote.

The researchers used the questions in a randomized controlled trial. Some citizens received phone calls asking the questions, others received the standard get-out-the-vote call and others did not receive a phone call. They found a dramatic result: People who were asked the planning questions were twice as likely to vote as people who were not.

There are other examples of scientific studies applied to campaign strategies. Rogers and a colleague at Yale University found evidence that messages which tell people high turnout is expected are more effective at motivating voters than messages which emphasize low voter turnout.

And a review of field studies in get-out-the-vote campaigns found that many of these strategies improve voter turnout, but tend to mobilize well-represented groups of people and miss under-represented groups – effectively widening disparities in the electorate.

Here at EBL, we think it is great news that social scientists are applying scientific methods to the art of political campaigning – especially where so much money is involved. And we hope it’s information that candidates use in their campaigns.

Is Red Meat Really Bad For You?

If you follow news about nutrition, you’ve likely heard the recommendations to avoid eating red and processed meats including beef, sausages and deli meats.

[Read more…]

You May Not Need a Vitamin D Supplement

More than half of adults in the U.S. take some type of vitamin each day – a number that has grown steadily over the past 20 years, according to the U.S. Centers for Disease Control. [Read more…]

How to Prevent Older Adults From Falling

Falls are a serious problem among older adults. Approximately one in every four people over age 65 fall each year, and 20 percent of those lead to a serious injury such as a broken bone or head injury.  Falls are the most common reason that older adults are admitted to the hospital. [Read more…]

Meditation Does Not Make You a Better Person

Many people think of meditation as a magic bullet that can improve our health, sharpen our focus, and make us better people overall.

But in reality, the evidence on meditation is mixed. According to reviews by the Cochrane Collaboration, there is little evidence that mindfulness can help improve health problems such as fibromyalgia, anxiety, depression, and neck pain. There are some credible data that demonstrate meditation can improve risk factors for cardiovascular disease. And research on brain function shows that meditation can reduce feelings of pain. [Read more…]

New Evidence Links Sugary Drinks and Obesity

You have, no doubt, heard the statistics about obesity in the U.S: Nearly forty percent of adults and nineteen percent of youth are obese, the highest rate the country has ever seen, according to the most recent data from the National Center for Health Statistics. We hear all sorts of reasons why people gain weight such as too many sugary drinks, not enough physical activity, and a lack of access to healthy foods. [Read more…]

What We Know About the Holiday Blues

Regardless of one’s religious affiliation, the holiday season often inspires feelings of warmth, joy, and belonging. But for some people, this time of year can evoke feelings of loneliness, stress and anxiety. [Read more…]

Does fasting work for weight loss and health?

obesity-300x225Every few years, a new diet becomes popular for its promise to help people lose weight and improve their health. Sometimes, these diets are based on sound evidence, and other times they are not. Often when new diets first become popular, there is not enough data to know whether they really work. [Read more…]

What we know about the holiday blues

depressionRegardless of one’s religious affiliation, the holiday season often inspires feelings of warmth, joy and belonging. But for some people, this time of year can evoke feelings of loneliness, stress and anxiety. [Read more…]

Skip to toolbar