The missing middle in evaluating fiscal governance

I have a confession: as someone who has done a lot of work on evaluation and learning at the intersection of open governance and global health, I am constantly jealous of my global health colleagues. Health has always been at the head of the pack when it comes to indicators and measurement. However, many of us working in governance (and especially fiscal governance, focusing on issues of budget transparency, trade policies, natural resource and extractive industry governance, and anti-corruption efforts) often fall back on responses like “well, what we do is much harder to measure” and “everything in governance is so much more dependent on context.”

But, during a chat with some global health colleagues about measuring progress in our respective fields, I had a chance to reflect on how much progress the fiscal governance field has indeed made in the last decade. This progress has helped to fuel some of work at R4D to push the boundaries of what’s possible in measuring progress towards fiscal governance outcomes (in plain English, the responsible management of public financial resources) in countries.

Lessons from phase 1 of the fiscal governance indicators project

One of the first things that struck our team when we began reviewing what already exists in the world of fiscal governance as part of our new fiscal governance indicators project was exactly that: big systematic changes — from open governance reforms to transparency in budgeting to whether civic space is expanding or shrinking in a country — are all things we can and do now track, sometimes quantitatively.

But the purpose of this project is not solely to understand what fiscal governance indicators currently exist. The primary goal is to find out what doesn’t exist, and to take a stab at filling this gap. You can read more about why organizations like the Open Society Foundations, the Transparency and Accountability Initiative, and the International Budget Partnership care about this goal here.

In this post, I want to share a little more about what our team has found to be missing in this fiscal governance indicators space. And what is missing — specifically that space between organizational activities and system-wide changes — came out loud and clear. If activities like advocacy campaigns and technical support to local CSOs are supposed to be catalysts that ultimately contribute to less corruption or better policies, how do we measure progress and track whether those activities actually do help bring about the big change? And, just as importantly, how do we know if they aren’t working?

Three Progress Indicator Gaps

Many outcomes could theoretically signal progress toward those ultimate outcomes that organizations working on improving fiscal governance are seeking — but a few key gaps stood out to us and our partners that seemed both critical and broad enough that they could fit into the theories of change (and action) of many organizations.  We share these priority gaps below, but we want to hear from you!  Do you agree with the outcome areas we identified? Does anything below excite you, concern you, or make you think “this is just plain wrong”? Please take a look at what we are planning and then let us know what you think at the end of this post!

1. Government Capacity, Incentives, Influence/Power (and/or Political Will?)

At the end of many theories of change in the fiscal governance field is some ultimate change — to policies and laws, to civic space, or to the political culture as a whole. And this change is influenced, if not fully caused, by an action taken by government actors or institutions. But understanding how and why and when we make progress toward those government actions is tough. For this reason, it seems like many organizations (ours included!) struggle to measure whether their actions targeting particular government actors or actions are moving progress in the right direction.

A first step in measuring progress toward government behavior change is to break down the “ingredients” that go into that behavior change. Based on our desk review and several interviews and focus groups, we have found a few elements that we think are key — and challenging — to measure:

• Capacity: What are the resources, tools, and skills that government actors need to influence the ultimate changes we want to see, and do they have them?
• Incentives: What are those motivations that we may not observe easily that are pushing government actors to maintain the status quo or change (good or bad)? These could include soft incentives (likability, altruism) or hard/concrete incentives (elections, monetary gain).
• Influence and power: To what degree does that actor have control (soft or hard) over the ultimate outcome? Like incentives, influence is multi-dimensional — it can mean power over domestic policy or at an international bargaining table, or something totally different.

Conspicuously missing from this list is political will. Should it be? After a lot of conversations with really smart people, our current belief is that political will may be (more or less) a combination of the three bullets above. But we want to hear from you if you disagree!

2. Capacity and Influence of the Civil Society Sector/Ecosystem

While some organizations seek to directly influence the behavior of specific actors, such as governments and corporations, others in the fiscal governance space (and beyond) focus on building the strength of ecosystems of actors working to reach ultimate outcomes such as improved government or corporate transparency. It is particularly challenging for such organizations to track the influence of their work on ecosystem strength — and therefore also on the ultimate outcomes they seek to affect. “Ecosystems,” amorphous groups of interconnected actors working toward a similar goal with or without coordination, are hard to define. Similarly, “strength” is a vague term that combines elements of capacity, connectedness, and influence. How does one measure changes in so many elements across a collection of actors?

Our research to date suggests that the key to measuring changes in the strength of ecosystems requires a clear definition and delineation of the focal ecosystem as well as unpacking the interrelated concepts of capacity, connectedness, and influence.

3. Message and Issue Framing

On the surface, this may seem like a sharp left turn. The previous two outcome areas are explicitly focused on specific targets: governments and/or civil society. This outcome, on the other hand, is about how successfully organizations connect with different audiences, from constituents to the general public to the private sector.

But one of the most frequently used techniques that fiscal governance-focused organizations use to connect with audiences is advocacy campaigns, to reframe messages, to saturate audiences with new and important information, and/or to create demand or pressure. While many advocacy activities have good performance targets (briefs downloaded, attendees at meetings), it is harder to assess how large-scale campaigns saturate, are used by target audiences, and change people’s minds and behavior. We may not be able to perfectly assess all of these components of progress — but we do think those measures of progress really need, well, progress.

Now it’s your turn

What did we miss?  Are there big “progress outcomes” that you think we have missed?  Are there aspects of the three outcomes above that you think we got wrong — or right?  And most importantly, does any of this sound like something you would use in tracking your progress working on fiscal governance (or other) issues?  We want to know what you think. Comment below and we’ll respond to you, or (if you’re on Twitter) reply to this tweet.