The Center for Evaluation Innovation started working with Luminare Group and the Johnson Center for Philanthropy several years ago on research that would help to frame what later would become the Equitable Evaluation Initiative (EEI). We talked a lot at the time about what it would take to get the philanthropic sector to embrace equitable evaluation. During those discussions, Jara Dean-Coffey, founder and director of the EEI, repeatedly referred to what she called the “aha” moment that would be key to getting people’s hearts and minds to embrace equitable evaluation and to position it at the center of their work.

I am a white cisgender woman. I have been an evaluator for 25 years. This was my “aha” moment.

During a break at a philanthropic conference I was attending, I ran into the head of a foundation. The conference had a strong diversity, equity, and inclusion (DEI) focus, and we were debriefing at the day’s end about what we had heard. The CEO wished the speakers had been clearer that foundations can make different choices, and that equity does not have to be part of every strategy. The CEO’s foundation, for example, had a program focused on the environment, but equity was not an explicit part of their strategic approach.

This conversation stopped me in my tracks. Partly because research shows that many environmental issues and policies — development, climate change, land use, pollution — have disproportionately harmful effects for populations that lack economic or political clout. How could an environmental strategy not focus on equity? 

But even more than that, it made me realize that I didn’t think equity applied to all of my work either.

It had been easy for me to engage with important conversations about equity happening across philanthropy. And a no-brainer to commit my organization’s time and resources to helping advance it. But it was less clear to me how equity applied to all of my day-to-day work.

Equitable evaluation gives us a framework for thinking about our work differently.

Equitable evaluation means aligning our practices with an equity approach — and even more powerfully, using evaluation as a tool for advancing equity. It means considering these four aspects, all at once:

• Diversity of our teams (beyond ethnic and cultural)
• Cultural appropriateness and validity of our methods
• Ability of our designs to reveal structural and systems-level drivers of inequity
• Degree to which those affected by what is being evaluated have the power to shape and own how evaluation happens.

While the concept of equitable evaluation makes sense from both a moral and a technical (methodologically valid) standpoint, it still can be hard to understand how it applies to any evaluation practice or to any evaluation, even evaluations of programs or strategies that do not have an explicit focus on advancing equity. This is where I was getting stuck.

I can do equitable evaluation when I am not evaluating equity efforts.

Many of my evaluation habits and routines over the more than two decades I’ve been an evaluator are based on orthodoxies that are widely practiced among evaluators, but not equitable. These routines are reinforced by professional norms in a field that is now, and long has been, majority white.

I needed to get more concrete. I needed to understand how to practice equitable evaluation regardless of the focus or topic. 

So my colleague Tanya (also white) and I tested it with our projects.

We applied it to one of our consulting practices — helping organizations to develop learning agendas. Most basically, being equitable in learning agenda development means identifying the learning questions we should be asking about equity. But also, it means being deliberate about the learning process itself, including:

• Examining who is defining what’s important to learn and for whom. (e.g., do grantees get to decide for themselves what questions are meaningful for them?)
• Answering questions with data that bring in perspectives normally unheard or discounted
• Attending to the assumptions built into learning questions that might undermine equity efforts (e.g., when we say “How can we design our capacity building supports so they increase grantee effectiveness?”, what do we mean by effectiveness?)
• Paying attention to who participates in sensemaking from data, and whether the conversation is designed in a way that accounts for power differentials.

We also applied it to an evaluation we had been working on that did not have an explicit equity focus. We were conducting a developmental evaluation of an initiative focused on strengthening the values, norms, and institutions (namely Congress) of U.S. democracy in a time of political polarization. We had not applied an equitable evaluation lens to this work. Applying one retrospectively, we realized we could have:

• Examined differential outcomes for populations, looking at how key performance measures would look when disaggregated by race. For example, what does it mean if public approval of Congress goes up among white men, while ratings for women and other racial groups decrease? Is that a “win”?
• Looked at structural drivers of inequity, or how the drivers in the system of representative democracy play out differently by race (or other aspects of inequity). For example, it’s not only that Congress doesn’t represent regular voters well, it’s that the interests of particular kinds of constituents are less likely to be heard and reflected.
• Ensured that data collection considered a broader range of perspectives.When we pressure tested the initiative’s strategy with experts, for example, we could have broadened whose perspectives we were gathering, such as experts who look at these issues from the angle of race and politics. We also could have asked experts to reflect not just on which approaches were important and likely to work, but also for whomand with what potential negative consequences for others.

The evaluation more consistently would have surfaced that solutions impact populations in different ways and are not universally beneficial. It made us realize that in our evaluation work, even if clients decide not to act on the insights generated, an equitable evaluation approach would push them to think through these issues and articulate why they are not.

Going through the exercise of how to be equitable in our work illuminated how our routines gloss over equity or only raise it periodically.

If I support equitable evaluation, I have to practice it.

Thinking about how equitable evaluation applied to our work, we also asked ourselves what we would do if the client explicitly did not want us to take this approach. Would we do it anyway and risk displeasing the client, or refuse the job to begin with?

My colleague Tanya said this question reminded her of a speaker we’d heard at a Grantmakers for Effective Organizations conference — Nikole Hannah-Jones, a New York Times reporter who has investigated why schools remain segregated. Understanding deeply how segregated schools negatively impact students, she and her husband had to decide whether to enroll their own daughter in a public school in their neighborhood that was segregated by both poverty and race, or use their middle-class privilege to get her into a different school. They chose to enroll her in the public school.

Hannah-Jones said she made this choice because it was important for her to live her values. She noted the hypocrisy of those who say they value equity, public schools, and integration, but when it comes to personal choices about their own children, they make decisions inconsistent with those values. They are willing to stand behind their values only up to the point where they bear personal risk. This is a privilege of choice that is available only to wealthier parents (watch her interview at the GEO conference).

Supporting equitable evaluation without practicing it is hypocritical. 

It also makes it harder for others to practice equitable evaluation by reinforcing professional norms and orthodoxies that are decidedly inequitable but still widely accepted and practiced. If we are truly committed to it, we have to be willing to bear potential risks to our revenue stream or client relationships.

Because equitable evaluation had not been part of all of my longstanding evaluation routines, I have had to think about it deliberately and work on shifting my practice consciously. I’m not there yet, and neither is CEI. We’re working on it, and are committed to more fully centering it in our work. This means applying it to both our field-building work and our evaluation practice. We look forward to sharing more about what our practice looks like as it continues to unfold.