Stories and Stats: How qualitative methods help us read between the numbers
[Editor’s note: This is the first post in the “Stories and Stats” blog series, which will share some exciting quantitative trends from the Transparency for Development (T4D) project, and dig into the stories behind the numbers. This post shares stories from T4D communities in Indonesia that demonstrate the importance of leaving room for local knowledge and creativity in international development projects.]
In Tanzania and Indonesia, Results for Development and the Harvard Kennedy School’s Ash Center for Democratic Governance and Innovation are evaluating whether community-led transparency and accountability efforts can lead to improvements in health outcomes. The intervention goes like this: we share local information about maternal and newborn health (MNH) with local communities (indicators, statistics, etc.); representatives from these communities — or community activists — participate in a series of meetings to identify barriers to better MNH; the community activists design and implement social action plans — operational, step-by-step plans to tackle a particular MNH problem; and we evaluate whether these action plans were effective.
This post shares some early insights into the most common types of actions we’re seeing in the Indonesian communities — and why it’s important to look beyond the statistical trends when trying to understand what’s happening on the ground.
Leave room for innovation
When deciding how to interface with the local communities, we adhered to certain design principles. “Non-prescriptive” was chief among our priorities — we firmly believe that we as outside researchers, our local civil society organization partners and even our facilitators (who often come from the same district as the villages they work in) don’t know the best, most effective solutions to MNH problems in individual communities. For this reason, we leave it to the community activists to decide what issues are most prevalent in their community and to develop social action plans that they think will be effective.
If this seems risky and difficult to evaluate, it’s because it is.
A traditional randomized controlled trial holds the intervention constant — administering the same vaccination, prescribing the same protocol, etc. — so any impact can be reasonably attributed to the intervention.
But a community-led effort, must be community led — and this allows for more creative, context-appropriate and effective solutions than any one-size-fits-all program. Instead of requiring community activists to follow a rigid protocol, we built a basic intervention based on providing information about health problems and facilitating community action planning to address those problems, but we allow for communities themselves to make many of the design decisions. This means we get much more variation than you would in a traditional RCT. We think that leaving room for community innovation and creativity is worth it — even if it requires our team to be more innovative in the evaluation design (We’ll share more on this in our upcoming evaluation design report).
Surprising variations behind one social action
During the social action planning meeting, community activists develop action plans to address the urgent MNH issues in their community. These plans might include building a new health facility in their village, initiating a socialization campaign to educate the community on proper MNH practices, fundraising to help women pay for transportation to the clinic, or any other number of activities.
As we reviewed the social actions coming out of our communities in Indonesia, we saw some statistical trends that seemed to indicate communities are choosing to undertake many of the same types of actions. However, upon closer examination of the “stories,” or qualitative data, we found that allowing community activists to drive the social action development resulted in some surprising variation and innovation.
Almost every group of community activists (97 percent of the communities) decided to tackle at least one barrier in their village by asking a health worker or government official to do something — advocating for reform. On the surface, this statistic makes the communities seem predictable. One could reasonably think, “If every community does the same thing anyway, why don’t we just prescribe the actions from the outset? Why complicate the evaluation by leaving actions open to communities, if they are all going to choose to do the same action?”
These questions would be valid if we looked solely at the statistics. But the 97 percent doesn’t tell the full story. Digging into the qualitative data, which gives us a an account of what has taken place on the ground, reveals amazing creativity and adaptability on the part of the community activists in navigating their particular context.
Advocacy actions undertaken by some communities were linear and followed steps and produced results that we might have predicted. In one village, the community activists identified the poorly performing midwife as a reason why women didn’t go to the local health facility to get MNH services. The community activists decided to go to the village head to ask him to replace the midwife and to make a few other minor improvements to the facility. The village head listened to the community activists concerns and enthusiastically agreed to help — he followed through with his promise and the midwife was replaced.
In another village, community activists identified inconsistent electricity at the local health facility as a reason why women chose not to go there. The community activists wrote a letter to the head of the health facility voicing their concerns and asking for a new generator. The head of the health facility then met with the community activists and promised to allocate funds for a new generator. These two examples are straightforward actions involving the community activists asking for something from a government or health official, and their requests being granted.
But many actions were more complicated and required innovation and resilience to overcome barriers. For example, community activists in one village wanted the health facility to post information about the cost of services so women would know how much to pay — presumably to avoid bribes and overcharging. The community activists started by recruiting the village head to help them get a meeting with the health facility head. Initially, the community activists’ request to install service cost information and to add more midwives was denied. But instead of giving up, the community activists identified a “middleman” who could help them get a meeting with the District Health Agency to apply pressure on the health facility head. This action is ongoing, so we don’t know whether the community activists have been successful, but it is a promising example of how communities identify and use the available resources to problem-solve and achieve their goals.
Other communities identified stakeholders outside of the formal health system and government as potential partners in their mission to improve MNH. One group of community activists decided to ask an existing community fund to coordinate with them in assisting pregnant women unable to cover the cost of transportation to deliver in a health facility. If the community activists identified women planning to deliver with a traditional birth attendant because she was unable to afford the cost of transportation, then the community activists would notify the community fund and they would subsidize the transportation cost.
These stories remind us that the health, political, social and cultural landscapes vary across villages. Communities themselves have critical insight into local resources and foreseeable challenges. Although the actions described here all fall into the category “advocating for reform,” we don’t prescribe this approach, because if we did, we would lose the innovative, collaborative and adaptable actions that respect local knowledge and context.
Hannah Hilligoss is a program and research assistant at the Harvard Kennedy School’s Ash Center for Democratic Governance and Innovation. She coordinates two fellowship programs and supports the Transparency for Development Project, among other initiatives under the Democratic Governance Program.
Photo © Results for Development
