Anyone who loves detective novels (like I do) winds up being fascinated by evidence. I remember discovering the Sherlock Holmes stories as a teenager, reading how the great detective systematically used evidence to solve perplexing crimes. Holmes followed the scientific method, gathering together a large amount of evidence, deducing several possible explanations, and then finding the one that best fits the facts of the case. As everyone knows, often the “evidence-based” solution was very different from what common sense told theo other people involved in the case.
In our efforts to solve human problems, we also search for evidence, but the solutions rarely turn up in such neat packages. Whether it’s a solution to teen pregnancy, drug abuse, family violence, poor school performance, wasteful use of energy, or a host of other problems – we wish we had a Sherlock Holmes around to definitively tell us which solution really works.
Over the past decade, efforts have grown to systematically take the evidence into consideration when developing programs to help people overcome life’s challenges. But what does “evidence-based” really mean?
Take a look at these three options: Which one fits the criteria for an evidence-based program?
1. A person carefully reviews the literature on a social problem. Based on high-quality research, she designs a program that follows the recommendations and ideas of researchers.
2. A person creates a program to address a problem. He conducts an evaluation of the program in which participants rate their experiences in the program and their satisfaction with it, both of which are highly positive.
3. An agency creates a program to help its clients. Agency staff run the program and collect pretest and post-test data on participants and a small control group. The group who did the program had better outcomes than the control group.
If you answered “None of the above,” you are correct. Number 3 is closest, but still doesn’t quite make it. Although many people don’t realize it, the term “evidence-based program” has a very clear and specific meaning.
To be called “evidence-based,” the following things must happen:
1. The program is evaluated using an experimental design. In such a design, people are assigned randomly into the treatment group (these folks get the program) or a control group (these folks don’t). When the program is done, both groups are compared. This design helps us be more certain that the results came from the program, and not some other factor (e.g., certain types of people decided to do the program, thus biasing the results). Sometimes this true experimental design isn’t possible, and a “quasi-experimental” design is used (more on that in a later post). Importantly, the program results should be replicated in more than one study.
2. The evaluation studies are submitted to peer review by other scientists, and often are published in peer-reviewed journals. After multiple evaluations, the program is often submitted to a federal agency or another scientific organization that endorses the program as evidence-based.
3. The program is presented in a manual so that it can be implemented locally, as close as possible to the way the program was designed. This kind of “treatment fidelity” is very important to achieve the demonstrated results of the program.
As you might already be thinking, a lot of issues come up when you consider implementing an evidence-based program. On the one hand, they have one enormous advantage: The odds are that they will work. That is, you can be reasonably confident that if implemented correctly, the program will achieve the results it says it will. A big problem, on the other hand, is that a program must meet local needs, and an evidence-based program may not be available on the topic you are interested in.
We’ll come back to these issues in later posts. In the meantime, I recommend this good summary prepared by extension staff at the University of Wisconsin. In addition, I’d suggest viewing this presentation by Jutta Dutterweich from Cornell’s Family Life Development Center, on “Planning for Evidence-Based Programs. And check out our web links for some sites that register and describe evidence-based programs.