How do I know if a program works? A “CAREful” approach

I was recently giving a talk on intervention research and I was asked: “How do I tell whether the evidence for a particular program is good or not?” I often talk with practitioners in various fields who are struggling with exactly what “evidence-based” means. They will read “evidence” about a program that relies only on whether participants liked it, or they will see an article in the media that recommends a treatment based on a single study. What should you look for when you are deciding: Is the evidence on this program good or not?

I came across a very helpful way of thinking about this issue in the work of educational psychologist Joel R. Levin. He developed the acronym “CAREful research,” which sums up what needs to be done when drawing conclusions from intervention research.

In Levin’s “CAREful” scheme, he identifies four basic components of sound intervention studies.

Comparison – choosing the right comparison group for the test of the intervention. Usually, there needs to be a group that does not receive the program being studied, so one can see if the program works relative to a group that does not receive it. A program description should explain how the comparison was done and why it is appropriate.

Again and again – The intervention program needs to be replicated across multiple studies; one positive finding isn’t enough.

Relationship – There has to be a relationship between the intervention and the outcome. That is, the intervention has to affect the outcome variables. That may seem simple, but it’s important; the program has to have a positive effect on important outcomes, or why should you use it?

Eliminate – The other possible explanations for an effect have to be eliminated, usually through random assignment to experimental and control groups and sound statistical analysis.

 Levin and colleauges sum up the CAREful scheme:

“If an appropriate Comparison reveals Again and again evidence of a direct Relationship between an intervention and a speciried outcome, while Eliminating all other competing explanations for the outcome, then the research yields scientifically confincing evidence of the intervention’s effectiveness.”

To see a good example of an evidence-based approach to intervention that reflects this kind of CAREful research, take a look at the PROSPER program, which takes a similar approach to youth development progams.

So when you are looking at intervention programs, “Be CAREful”: Applying these four criteria for good research can help you decide what works and what doesn’t.