Testing behavior change communications to reduce family-child separation in Cambodia

[Editor’s Note: This blog is the first in a two-part series about Rapid Feedback Monitoring, Evaluation, Research, and Learning (RF MERL), an innovative monitoring and evaluation mechanism the U.S. Agency for International Development (USAID) — and what we learned from our first completed engagement. This blog shares the evidence we generated on the two behavior change interventions we tested to reduce unnecessary family-child separation in Cambodia. In the second blog, we will discuss what we learned about applying the RF MERL approach itself within USAID-supported activities.]

By Morgan Benson, Abigail Conrad (of Abt Associates) and Christina Synowiec

Cambodia has experienced a recent, rapid rise in the number of residential care institutions (RCIs), placing children at risk for unnecessary separation from their families. Local communities and international donors often have good intentions in supporting these RCIs, but are unaware of the potential harms or community-based care options available. To that end, USAID launched Family Care First (FCF) Cambodia in 2014 to discover and advance transformational solutions to reduce the number of children growing up outside of safe, nurturing and family-based care.

When the initiative began, there was relatively little known about how to create behavior change among communities and donors to move away from these institutions and toward family-based care. In response, FCF partnered with Rapid Feedback Monitoring, Evaluation, Research and Learning, also known as RF MERL, to generate evidence on the effectiveness of behavior change communication activities through FCF Cambodia.

RF MERL is a rigorous, flexible approach to generating timely information for programs to adapt and improve activity design throughout implementation. Supported by USAID, RF MERL recently concluded a 1.5-year-long engagement with the FCF Cambodia initiative.

Embedding rapid feedback into FCF Cambodia design and implementation activities

The RF MERL Consortium partnered with Cambodian Children’s Trust (CCT) and Friends International (FI), two Cambodian nonprofit alternative care institutions, to experiment with approaches to create behavior change among communities and international donors. We co-developed a theory of change with these partners, and identified assumptions for testing that had both high potential for impact and uncertainty around their optimal design. This process led us to the question: what activities at the community and donor level would result in the behavior change of supporting family-based care over RCIs?

Testing behavior change communication campaigns in Cambodian communities

RF MERL kicked off the engagement with formative research in potential target Cambodian communities for CCT’s social work and behavior change activities. We wanted to understand the types of campaign messages and communication channels that would resonate with community members. We also worked alongside CCT to conduct rounds of lean testing, a method to quickly and iteratively test basic prototypes, focused on campaign messages and images. Based on the findings, we co-designed a behavior change communication campaign using the messages and communication channels that resonated with our target communities. We then moved into a feedback experiment, or tests of one or more promising intervention options, to gather rapid feedback to iteratively improve intervention design and to select which options should be implemented on a larger scale.

We compared households in communities exposed to two different interventions: 1) CCT’s social work model plus a social behavior change communication (SBCC) messaging campaign, and 2) CCT’s social work model alone. In order to understand how the SBCC campaign impacted households’ knowledge and attitudes about RCIs, we used a mixed-methods evaluation, described below:

What we learned:

1. Respondents in villages that received a formal messaging campaign along with CCT’s existing social work were slightly more likely to be aware that RCIs can result in psychological or emotional harms in children, and less likely to report that they may actually send their child to an RCI in the future. This led to recommendations to improve the campaign before further testing, such as ensuring that village chiefs and other key influencers understand the issues and support families in engaging with the messaging.
2. Violence in the household and if one or more parents lives outside of the household are key risk factors for children being sent to RCIs. This finding supported social workers to target their work in communities most affected by these factors. 

Testing behavior change communication campaigns targeting international donors

RF MERL engaged in a similar process of formative research with FI to understand the types of campaign messages and communication channels that would resonate with international donors contributing to Cambodian RCIs. We then moved into a Feedback Experiment focused on comparing social media platforms for the campaign.

We compared audiences exposed to two different online channels: 1) ads promoted through Facebook Ads, which display in Facebook users’ newsfeeds; and 2) ads promoted through Google Display Ads, which display on websites and mobile apps in Google’s network of websites and apps. These emerged as important channels in the formative research and ones that FI wanted to compare given their existing communications activities. The two ads we tested, developed by FI, served as the “treatment” ads in the experiment.

To assess the effectiveness of the ads, we evaluated the impact of the ads on donors’ perceptions of RCIs and likelihood of donating and volunteering, as well as the costs to FI of posting the ads. We did this in two phases. During phase one, we tested ads on Google and Facebook to answer the questions: Is there a difference between the effectiveness of the ads on the two platforms? We had 1,494 Facebook respondents and 1,518 Google respondents who answered a three-question survey after seeing one of the three ads: A, B or control. Since the answer to our phase one questions was “no,” we asked the following question in phase two: What is the cost-effectiveness of running the ads on Facebook versus Google? To answer this question, we gathered data through web analytics on ad performance and costs as well analytics from FI’s website.

What we learned:

1. Digital communication can have an impact. Informational ads on Facebook had a small, detectable effect on viewers’ attitudes about RCIs. However, neither Facebook nor Google influenced respondents’ likelihood of future donation or volunteering behavior.
2. A static ad is not enough to result in behavior change. Instead, we recommended looking into video ads or interactive campaigns to increase engagement with the key messaging and ideally lead to more impact on viewers.

These insights were critical to help CCT and FI develop, test, and revise approaches to their behavior change communication campaigns. As a result of the findings, both implementers are now revising their targeting and messaging approaches, in addition to further refining the channels for delivering the campaigns. To learn more, see the two briefs we produced, each focused on one of the two groups we aimed to impact.

***

About Family Care First Cambodia
Working with the Royal Government of Cambodia and private donors, the United States Agency for International Development (USAID) launched the Family Care First (FCF) Cambodia initiative in 2014. Guided by the collective impact model for structured, multi-sector collaboration, one of two cooperative agreements was issued to Save the Children for the Cambodia Families are Stronger Together (FAST) project. The FAST project includes more than 25 implementing organizations, including Cambodian Children’s Trust and Friends International, and seeks to develop a comprehensive care system in Cambodia to prevent family-child separation and promote the reintegration of children from RCIs to families. The FCF partnership with RF MERL was supported by USAID’s Global Development Lab (Lab)/Center for Development Innovation (CDI) and the Bureau for Democracy, Conflict and Humanitarian Assistance (DCHA)/Center of Excellence on Democracy, Human Rights and Governance (DRG)/Empowerment and Inclusion Division (EI), and USAID/Cambodia.

About Rapid Feedback Monitoring, Evaluation, Research, and Learning
The United States Agency for International Development (USAID) signed a cooperative agreement with a consortium of four organizations including Results for Development (R4D), Abt Associates, Mathematica Policy Research, and the Notre Dame Initiative for Global Development in September 2015 to implement Rapid Feedback Monitoring, Evaluation, Research, and Learning (RF MERL). RF MERL is an innovative initiative under the Monitoring, Evaluation, Research, and Learning Innovations (MERLIN) Program through the U.S. Global Development Lab in partnership with the USAID Bureau for Policy, Planning, and Learning and the Bureau for Global Health.