From the e-book, How to Show Off Your Wicked Strong Sexy and Incredibly Effective Nonprofit, available on the website of the Effective Communities Project
Steven E. Mayer, Ph.D.
Is your nonprofit program effective? And what does that even mean?
Every nonprofit manager, CEO, and board member gets asked this question, “How do you know your program is doing any good? How do you know it’s effective?” These are usually difficult to answer.
We believe being effective means doing good in the ways suggested by a nonprofit’s mission statement – and being able to show it. A good mission statement makes clear just who the organization intends to benefit; it names intended beneficiaries and intended benefits.
For example, one nonprofit’s mission statement might be, “To address hunger in our community, we serve nutritious meals to those who can’t afford them.” This means that poor and hungry individuals in the community are this nonprofit’s primary intended beneficiary. Doing good and being effective means making progress in your pursuit of ending hunger in your community by serving nutritious meals to those who can’t afford them – and being able to show that you’re making progress.
“Who benefits and how” is the meat and potatoes question of nonprofit program development and evaluation. This question may not be on top of everyone’s list. Some want to know if you’ve lived up to the terms of the grant. Some want to know more about cost, and the proportion of budget spent on administration or fundraising. Some are satisfied to know that your organization is growing. Those may be legitimate variations of the theme of effectiveness, but to us they are secondary.
To us, the central focus of effectiveness is brought by the question, “Are you being genuinely useful to your intended beneficiaries in the ways suggested by your mission statement? Can you show the ways you’re making your mission a reality?”
It’s an important question because deep down this is what donors and allies really want to know. They want to know they’re contributing not just to a good cause, but to an organization that’s delivering.
We’re not saying you must show you’ve solved the community’s hunger problem, but without evidence that you’re delivering on the challenge of serving nutritious meals to poor and hungry people, the organization is banking simply on a promise and the good will of people who care. With evidence, it’s more likely you’ll gain support to build your nonprofit’s organizational capacity to make even more progress.
Progress counts. In short, a program is effective when it can show it’s making progress on its mission. And just as there’s lots of nuance and wriggle room in the way the mission is stated, there’s also nuance and wriggle room in the way a nonprofit can show this progress. But if this nonprofit is unable to show even a lick of progress, it can’t be considered effective.
Some might say this is a dodge, an escape from demands of “real proof you’re ending hunger.” But our vision of nonprofit effectiveness respects mission and the ongoing cumulative work required to achieve it. We want to make “improved effectiveness” rather than “proof of long–term impact” the goal of nonprofit development and evaluation. Rather than seek the scientist’s version of proof or the investor’s version of bottom line return on investment, we prefer to seek evidence that progress is indeed being made in delivering intended benefits to intended beneficiaries.
We’ll have more to say about nonprofit missions and evaluating for effectiveness in Chapter 2 – Dirty Little Secrets of Mission Effectiveness. And more about evidence in Chapter 5 – Sexy Evaluation Strategies.
Beneficiaries and benefits: Who gets the Love, Money, and Power?
OK, that heading was written to feed the bots that scour the internet looking for buzzy keywords. To bring some clarity to the concept of who benefits and how, we offer this simple breakdown of four major groups of intended beneficiaries of nonprofit activity.
Working to benefit individuals is the core of the charitable sector’s work, at least historically. Delivering services to address the basic human needs of individuals is the traditional work of “human service” or “social service” agencies. A “hierarchy of needs” famously described by Abraham Maslow parallels the gamut of charitable programs developed in the last 150 years.
Maslow’s hierarchy starts at the most basic and necessary level, the physiological needs for survival; this includes the need for food, water, air, warmth, clothing, rest. The hierarchy progresses to needs for safety, security, stability, housing, work, some measure of financial security. The US is particularly rich with nonprofit and public programs that serve individuals in these areas.
Higher in Maslow’s hierarchy are the love and belonging needs for acceptance, belonging, friendship, affection, support, love. Higher still are the self–esteem needs for skills, knowledge, confidence, direction, accomplishment, dignity. And highest is the need for self–actualization, including the need for meaning, purpose, fulfillment, creativity, dignity, achieving one’s full potential. Thousands of nonprofits and millions of self–help books, websites, and commercial services are designed to address these needs.
Intended benefits. The intention is that there is indeed improvement in the quality of lives of individuals in ways suggested by the mission of the program – essentially in a reduction of urgent need and an increase in prolonged vitality.
- In programs with a mission to address physiological needs for survival, the intended benefits are a reduction in risk of imminent misery or death – and greater chance of comfort, health, and survival.
- In programs with a mission to address needs for safety, the intended benefits are reductions in fear, instability, joblessness, homelessness, and poverty – and increases in comfort, stability, productive work, satisfactory shelter, and some measure of financial security, and improved prospects of surviving and thriving.
- In programs with a mission to address higher–order needs, the intended benefits are reductions in isolation or feelings of worthlessness and unhappiness – and increases in purpose, dignity, and meaning.
Societies since forever have developed systems that are geared to addressing the same needs that traditional charities address at the individual level – but they do it “systemically.” Think of the food distribution system, the clean water distribution system, the educational system, health care delivery system, criminal justice system, wealth building system, electoral system – just to name a few that we’re all a part of even though we can’t see them in their entirety.
Systems operate at all levels, from local to international, typically with different mandates and authorities at each level. They are formal, with policies, and informal, with habits and values. They are designed and operated by humans, and they therefore operate with flaws (some serious). They therefore can all be improved.
It is the work of many nonprofits to improve the performance of these systems so that they work better for more people. Activists in the system change world often speak of “moving the needle,” referring to a metric (such as the one on your car’s dashboard showing level of fuel in your tank) allegedly showing the performance of a mission–critical part of the overall system. With social systems as complex and out–of–centralized–control as they are, the challenge is to identify the mission–critical parts of the overall system, and shape and improve the just–right metrics once we’ve agreed on the goals we have for these systems.
Intended benefits. Working on systems ideally allows the relevant performance needles or metrics to move to ever more positive territory. The work of system improvement (aka reform or transformation) typically intends progress in…
- Building ever more capable organizations with supportive cultures and allies that want to make progress.
- Developing leadership, relationships built on trust, creating networks, building clout, gaining access to greater authority.
- Advancing promising solutions, from idea to testing, from upgrading to scaling up, with protocols for training, monitoring, and accountability.
- Accumulating a variety of resources – money, information, technology, blueprints, wisdom – and using them smartly
The needs documented by Abraham Maslow that are addressed in programs serving individuals are also addressed in programs serving communities. And while the category of Communities is a diverse set, the similarity is that programming is directed to them as a group – a concert, a curriculum, a conference, etc. A community can be defined by:
- Demography, ethnicity or cultural group, such as youth, Italo–Americans, the elderly, or classical music lovers.
- Geography, such as a neighborhood or watershed, or any political jurisdiction like county, state, or nation.
- Audiences, such as those of arts and culture organizations, intended audiences of public health messages, or those viewing your organization’s website.
- Affinity, such as those experiencing a community–wide condition or experience, such as victims of any disease entity or man–made disaster, or the Class of 2001, everyone who’s seen Inglorious Basterds, or members of the state association of state associations.
Intended benefits: The intended benefits of community–serving programs are similar to programs serving individuals. Progress is typically noted as a mission–relevant success rate for the group or community being targeted. Common examples could include satisfaction rate among audience members, income level in a neighborhood, mortality rates, wealth rates, etc. Policy organizations in many arenas have been advancing measures of a community’s “quality of life” for many years now.
Working to benefit organizations – nonprofit organizations in particular – is the work of grantmaking foundations, individual donors, private investors, and networks of like–minded organizations. While these entities may well hope that individuals and communities will ultimately benefit, the primary and most immediate beneficiary is the recipient organization.
Intended benefits. Recipient organizations can benefit by becoming stronger and more capable in carrying out their missions. These gains in “organizational capacity” can include:
- Upgrades to administration including improved board governance, management, policies and procedures, staffing, budgeting;
- Upgrades to resource development including improved communications, fundraising and endowment building, information and technology, infrastructure;
- Upgrades to community linkages including strengthened connections with other key organizations and networks, strategic partnerships, and clarified role in the community; and
- Upgrades to programming to become a stronger learning organization, with stronger connection between understanding consequences of program activity and strengthening the program to create more better benefits to beneficiaries.
For more on organizational capacity click here (soon) and on whether foundations or nonprofits should get credit for ending hunger and such, click here (later.)
These four categories present the four essential intended beneficiaries of nonprofit programs – and the essential benefits intended for each. These intentions express hopes and dreams, as well as blood, sweat, and tears. Many things, especially life itself, can get in the way and spoil the effort. Resources are short and staff and volunteers are overworked. The problems created by contemporary society seem to be accelerating, and the struggle for making progress on missions seems increasingly uphill. If progress is to be made, progress has to be funded.
If good things happen but no one notices, does it count?
If good things happen but there’s no one there to notice, do they count? Can you still get credit for them? Answer: Probably not. There are several things to consider in your pursuit of showing off your wicked strong sexy and incredibly effective nonprofit by telling the story of your progress toward your mission.
Unnoticed, unclaimed, unloved. In truth, most nonprofits have multiple beneficiaries, and for each there are multiple intended benefits – often unstated and unrealized. Even though they typically get paid to deliver for their primary intended beneficiary only, most nonprofits impact other beneficiaries as well. But unless a nonprofit can track what’s happening with other categories of beneficiaries these benefits go unnoticed, unclaimed, uncommunicated, unloved, and unfunded.
For example, a soup kitchen may intend primarily to serve nutritious meals to individuals, but it might also participate in advocacy efforts to improve the food distribution system, partner with and assist other organizations in their efforts, or provide nutrition education to segments of the community. It can easily happen that the nonprofit is paid for work with its primary beneficiary – individuals receiving meals – but not paid for the other work, for which it’s forced to volunteer. I’ll bet it explains a lot of the burnout we see in the sector.
There’s a potential success story to tell about each of your intended beneficiaries. And there are potential upgrades to the design of your programs to deepen your impact with all of them. So when a nonprofit serves multiple beneficiaries it should try to take credit for all it does, and to muster evidence to back it up. Too often, nonprofits take credit for too little, and lose out on opportunities to upgrade and move forward with additional support.
Unrealistic demands. It’s commonplace that funders expect a nonprofit to reach beyond its grasp into areas where it has no control. For example, an alternative school will be asked to prove that its graduates go on to enter esteemed professions later in life – proof that even Harvard can’t provide. Or a chamber orchestra will be asked to prove that audience members go onto become lifelong lovers of their music. Or an organization supplying medicines to displaced persons along the border will be asked to show it is stemming the flow of refugees. As underfunded as most nonprofits are, more realistic demands for evidence should be the basis of more productive discussions.
Realistic demands. Just saying “We save lives” or “We develop leaders for tomorrow” or “We benefit the homeless” doesn’t make it so. You can state your intentions to save lives, develop leaders or benefit the homeless, but your stakeholders need you to show evidence that you’re working on those intentions and making progress. This doesn’t strike us as an unreasonable demand. It’s showing “signs of progress,” or “early signs of impact” that’s important.
So for example, if you’re serving meals, can you show (perhaps in addition to number of meals served), progress in trying out new sources of nutritious meals, partnering with others to extend food supply into communities that could be called food deserts, or strengthening the board of your organization so it can better help you make these connections?
A jury of your peers. An exercise to expand your sense of what’s possible to response to demands for “proof your program works.” Pretend your Executive Director is hauled before an “Evaluation Court” and charged with “operating an ineffective program.” To avoid conviction, imagine the kinds of evidence that could be mustered in defense, and ideally show the opposite – that in fact this program really is effective? In this exercise in make believe, you’re encouraged to think more broadly about effectiveness and evidence. What sort of “witnesses” could offer “testimony” that “makes your case” for this program’s effectiveness? Pretend further that this case must persuade a “jury of your peers and stakeholders.” Maybe there’s even opportunity for cross–examination. Pretend you have a year before going to trial. That should give you time to gather witnesses, prepare testimony, and develop a convincing case.
Not scientific? OK, it’s not the stuff of controlled laboratory experiments, or randomized field trials with control and experimental groups. But it is empirically sound. And what we’re doing here is substituting the rules of a courtroom for the rules of a science experiment. With this courtroom approach we can have multiple arguments, multiple bottom lines for beneficiaries, multiple lines of evidence speaking to benefits received, multiple voices from multiple points of view. Yes, it’s the antithesis of a social science experiment, with its independent variable (a particular program, allegedly delivered in the most standardized of fashion) administered to experimental groups and control groups chosen to be as homogeneous as possible, such as lab mice or college sophomores to make more likely a finding of “statistical significance” on a single measure that allegedly expresses success. This model demands conditions that do not exist in the real world of complicated human beings and ever–changing conditions within and between groups, and ever–changing administration of the program where nothing is standardized, yet randomized double–blind experimental trials are considered by many to be the gold standard for evaluating a social program intended to improve real–world quality of life. And since many on the nonprofit’s board and its funders’ boards have taken at least a few social science courses, the social science model prevails. We encourage the social sciences to go back to their roots: empiricism.
Show me = Empiricism. “Show me” is the heart of the empirical tradition, which in turn is the heart of the scientific tradition. In olden times, as historians would tell us, when testing whether a new method of planting corn, for example, was better than current practice, they’d try the new method next to the old one and invite all the nearby farmers to see how it was going. If license plates can be believed, Missouri is “The Show Me State.”
So flash forward to the case in hand: “Show us all the signs of your progress – let’s see what you’ve got.” For example,
- Can you find experts that can speak to the merits of your program design? Are there people from National who can testify you follow Good Practices?
- Can you count the number of people receiving meals, the pounds of food served, how much is fresh and nutritious, the amount of waste, the cost/meal, and the number of people turned away? Some of these counts can be constructed from billing traffic counts, testimony from providers.
- Can you interview people on site, or afterwards, and ask them about their experience? Can they express things on a 5–point scale indicating how much safer or less hungry or more secure they now are?
- Can you show an album with photos of significant events, important documents, or letters of appreciation? Can your database person put together some cool graphics from data already collected?
- Can your meeting notes be examined to reveal where you are now on some critical dimensions compared to where you were then?
- Can you make comparisons for any of the above against the evidence from last year, or against the industry average, or against your standing from the beginning.
- Can you engage the best listener and communicator to help you formulate the elements of a good case, keep track of the organization’s ability to locate evidence, and plan productive additions to it?
The M–word. The word “measurement” can be intimidating. But measurement is nothing more than making comparisons. That’s it. It doesn’t have to use numbers; we “take the measure” of a person or situation all the time. We size them up, we take a reading, we judge their qualities, we gain an understanding of them – we evaluate them. We invite the neighbors over to take a look and judge for themselves. Measurement is a human activity. It’s familiar and meaningful, and we all do it. “Taking measures” can be made engaging, jointly owned, productive, and useful. Just naming the categories for which simple but meaningful counts can be made helps to advance your communications agenda.
What’s to be compared? The state of the organization’s ability to produce today compared to last year. The number of people who experience X versus those who experience Y. The number of good things you can say when they leave compared with the number of bad things (maybe from an acceptable list of signs of success). Over time you’ll learn to better lift up what’s important, getting better at and observe, note, compile, and aggregate evidence to show your organization’s progress towards mission.
Then comes the E–word, Evaluation. See the word “value” in there? Evaluation is the process of making meaning from your measurements. What’s to be said from a table of numbers, or a photo collection, or a graph?
The data do not speak for themselves. And without belaboring the obvious, data don’t speak at all. Meaning has to be made from them, by humans. We guarantee no two people will draw the same conclusions, insights, implications, or recommendations from even one page of evidence.
One thing about measurement is, whether with categorical comparisons of frequency counts, or numerical analysis done on interval scale data, it’s not as objective as purported. Deciding what to count, compare, photograph, make notes of etc are all human choices, and human interpretation of the collection of data is required, even though numbers people like to deny this.
Goal: Improve, More Than Prove. We believe that the point of evaluation is to “improve rather than prove” the value of work done by nonprofits. The notions of “outcomes,” or “results,” or “impact” are limiting – they sound final, as if denying the continuing stream of behavior and events that happen in the real, uncontrolled and unstandardized world. At the very least, we could encourage “outcomes to date,” or “results so far,” or “early signs of impact.” Evaluations done with an Improve not Prove mentality should dig up lots of opportunities for improvement, upgrading, development – all to be incorporated into the next funding requests.
“Effectiveness” is a construct – make it work for you. Effectiveness is a “construct” – an abstraction, not directly observable, but understood from a variety of different signs or markers taken from different points of view. “Health” and “Wealth” and “Success” are constructs, too, meaning there are many different measures that, once rolled up into a whole, give an understanding of what’s meant.
What does “effectiveness” or success look like? What would indicate “effectiveness” in the case of your nonprofit? Those are excellent questions worthy of a board or staff retreat. The answers depend almost completely on your mission, the culture and context of your organization, and what you’d like your stakeholders to understand. If your program focuses on reducing infant mortality, use measures of infant mortality that your program wants to influence. If it focuses on increasing high school graduation rates, use measures of high school graduation your program can relate to. If your program has no intention of increase high school graduation rates, then don’t claim you can and don’t allow funders to say “You were supposed to increase high school graduation” when you weren’t. Instead, make your ways of showing “effectiveness” flow from your own mission, choose measures that relate to what you are actually doing, and own them. Make “effectiveness” yours.
It’s a nonprofit’s responsibility – and opportunity – to unpack Effectiveness in all its grandeur, complexity, and value to the world. . Acknowledge you’re making a case that shows progress toward your mission, or reveals a positive trajectory of past, present, future. Use language that speaks to you and to your different stakeholders. Make sure the story of who benefits gets told.
Steven E. Mayer, Ph.D. [rev. 8/13/2019]
(c) Effective Communities LLC