What can we learn from participants in transparency and accountability interventions?

By Stephen Kosack, an associate professor of public policy and governance at the University of Washington, Seattle, and a senior research fellow at the Harvard Kennedy School Ash Center for Democratic Governance and Innovation

Why do we sometimes hear that transparency and accountability programs empower citizens and community organizations to measurably improve the responsiveness and effectiveness of public services (here and here, for example), and sometimes we hear the opposite (see here, here, and here, for example)?

The effectiveness of development programs is generally assessed on objectively measurable outcome indicators: school attendance rates, test scores, per capita incomes, infant mortality rates, and other objective, comparable measures of impact. But perhaps we can also learn something about the implications of these programs from the experiences of those who participate.

In 2015 and 2016, several thousand people in 200 randomly-selected communities across four regions of Indonesia and Tanzania attended a series of facilitated meetings — as part of a mixed-method research project called Transparency for Development, to explore whether citizen engagement can improve maternal and newborn health outcomes. These meetings were carefully co-designed designed with civil society partners to encourage participants to work together to improve their maternal and newborn health care (see here, here, and here).

Like other evaluations, ours will assess the impact of this program on objective measures of maternal and newborn health and health care. But those improvements depend on the civic engagement of diverse participants in diverse communities. Thus, we are also attempting to understand the experiences of participants in this program, and the similarities and differences in their experiences. We are doing so from several perspectives, among them: 1) systematic qualitative observations of meeting discussions at the beginning and end of the program in 81 of these communities (41 in Indonesia and 40 in Tanzania), and 2) interviews with the people who participated before the first program meeting and immediately after the last in 32 communities (15 in Indonesia and 17 in Tanzania).

What are we learning so far? In a nutshell:

1. Many people were willing to give the meetings and the civic activities they designed in them a try.
2. Most, although far from all, seemed to think that they were getting somewhere.

Many people were willing to try

In most communities, many people showed up. Despite not receiving compensation or resources, an average of 14 people attended meetings over roughly three months of the program. Interviews and meeting observations suggest that in these meetings they engaged in sustained and generally self-directed discussions about problems with their maternal and newborn health care and deliberations about how they might alleviate those problems. Participants in almost all communities were also subsequently willing to try at least some of the civic activities they had planned. Here are a few indications:

It was rare for participants in meetings to sit quietly. At the first meeting, which was designed to provide space for participants to discuss and deliberate on information the facilitator presented on the state of maternal and newborn health care in the community (based on international standards and a survey of the community and its experiences of maternal and newborn health care in the previous weeks) and then to decide on the problems with that care that they would like to focus their efforts on, only 7 percent of participants made no distinctive contributions to the discussion. In the final meeting, when the facilitator asked participants to reflect on the progress they had made over the previous three months and to plan for the sustainability of their efforts once the facilitator had left for the final time, only 17 percent of participants made no distinct contributions to the discussion. In both countries, a majority of those who spoke in the discussions were women.

Generally, although not always, participants seemed to make decisions themselves rather than relying on the program facilitator. At several points in the discussions, observers also noted when participants relied on the facilitator or when their deliberations were controlled by the facilitator. In almost all communities, observers noted that in the majority of these discussions, the facilitator let those attending decide for themselves which problems to focus on or which approaches to take in resolving them. It was also uncommon for participants to rely on or defer to the facilitator: observers noted participants relying on the facilitator in most of the discussions in the first meeting in only five of the 81 communities, and most of the discussions in the last meeting in only 10 communities.

Participants were generally willing to try out what they planned, and to keep trying for several months. At the end of the program, approximately three months after the first program meeting, participants in almost every community still appeared engaged in at least some of the civic activities they had planned in the meetings to try to improve their maternal and newborn health care. How do we know? That’s a question we are exploring from a number of perspectives. But one indication from the meeting observations is that in all but one of the 81 communities, participants in the last program meeting engaged in detailed discussions reflecting on progress they had made and challenges they had faced in community activities that they discussed trying since the previous meeting about a month earlier — activities to improve knowledge or use of maternal and newborn health services, the responsiveness or effectiveness of health providers or officials, or to otherwise improve their care.

Was It Worth It? Optimism and Civic Efficacy

These meeting observations and interviews with individual participants also suggest that most thought their efforts were paying off. But far from all.

First, in discussions in the final program meeting, participants in 55 of the 81 communities where observers attended meetings seemed optimistic that the approaches they had designed and attempted outside the meetings were sustaining meaningful improvements in their maternal and newborn health care; this includes seven communities where participants seemed to begin the program skeptical that they would be able to improve their care. But in 26 of the 81 communities where observers attended meetings, participants seemed skeptical that they would be able to sustain improvements in their care. And context mattered: 21 of these 26 communities were in Tanzania.

We see something similar in interviews. In identical interviews with individual participants before and after the program, most seemed confident in their capacities to achieve the sort of improvements in their community’s public services that the program would encourage them to try, and significantly more told interviewers after the program that they were more confident in those capacities. These increases remain statistically significant even after we controlled for participants’ age, gender, education, community context, and used anchoring vignettes to adjust for differing interpretations participants might have of what it means to improve their communities.

Yet confidence was not universal, and, again, context mattered. In Tanzania, 14 percent of participants reported higher civic efficacy at the end of the program than before it began, but 19 percent reported lower civic efficacy. By contrast, in Indonesia, 46 percent of participants reported higher civic efficacy when the facilitator was leaving the community for the final time. Only 7 percent reported lower.

In addition to context, when we looked more closely at who seemed to lose confidence, we found that the people who said they were less confident at the end of the program tended to be those who did not participate as much: who spoke little or not at all at meetings and who did not engage in planned activities outside the meetings. If a person spoke an average amount or more, they tended to report higher civic efficacy at the program’s conclusion.

The Bottom Line

Overall, these observations and interviews suggest that a transparency and accountability program can encourage civic participation in a large number of diverse communities — but not everywhere.

Our team is now digging deeper, with in-depth ethnographic studies of what it was like for participants and their broader communities to be a part of this program, statistical comparisons of objective indicators of their maternal and newborn health and health care to another randomly selected group of communities who were not offered the program, and analyses of contextual differences that may have influenced differences in the program’s effects. More on them all soon.

If you want more details on our findings about participation and civic efficacy, you can read more here in an early draft of a working paper from our team.