Adapted from my course on Nonprofit Program Development and Evaluation, part of the on-line Nonprofit Management Certificate Program offered by Johns Hopkins University.
How can we evaluate a program’s effectiveness? I translate the question as, How can we collect good evidence of our program’s effectiveness. Evaluating effectiveness is all about “collecting good evidence.”
I use the term “evidence” rather than “data” to reinforce the notion that data, to be useful, must be persuasive, and support a use more than a theory. Supporting a theory is the job of science; supporting a use is the job of evaluation. In science we ideally construct experiments on comparison groups that we control; but in the real world such control is impractical to achieve. Instead we have to rely on other methods (variations of interviews, surveys, and time series analysis of performance data), and we rely on them to answer more immediate questions on value, performance, and opportunities for improvement.
I like to promote the notion that conducting a compelling evaluation inquiry more closely follows the rules of a courtroom than the rules of a scientific laboratory. You want to find and present evidence that makes the case for effectiveness, and informs a particular decision.
For example, if the purpose of the evaluation is to decide whether or not to support a particular program or effort, the evidence must allow the reader/stakeholder to lean one way or the other and make an informed decision — a very different purpose than trying to prove the efficacy of a particular approach, a far more daunting task requiring the ground rules of science. Moreover, the inquiry should be designed to reveal opportunities to improve the program’s effectiveness, not just give a thumbs up or down.
The more useful work of evaluation in most contexts is to allow us to improve, more than prove, the value of a program.
If you (in your donor role) are wondering how much to support a particular program, and you want to base your decision on an evaluation, you could look for or ask for evidence that the program is actually benefiting those intended in the ways intended, and for recommendations on what would make it better.
If you (in your recipient role) are searching for donor support, you could report on the efforts that have brought you to this level of effectiveness (as seen in the evidence), and the efforts that will take you farther in making your mission come even more true.
# # #
Steven E. Mayer, Ph.D. / Effective Communities Project / November 5, 2012