The Challenge
As global education efforts move away from access and toward learning as their key priority, the ability to assess learned skills like basic literacy and numeracy becomes more important. Many countries produce data on student learning, mainly from national in-school assessments. Given the large number of out-of-school children and high absenteeism rates, such learning assessments have been criticized for not being representative of a country’s entire population of children.
The Opportunity
Citizen-led assessments of basic math and reading skills can provide accurate data on levels of knowledge, leading to informed budget decision-making.
To date, organizations in six countries — India, Kenya, Uganda, Tanzania, Mali and Senegal — have conducted nationwide, citizen-led assessments of basic reading and math skills. As part of the work, local volunteers were trained to administer simple numeracy and literacy assessments in households across the countries. Interest in the model is spreading to other countries as well.
Our Results
R4D conducted an evaluation of those efforts in order to inform how these types of assessments might evolve and expand over time. The evaluation aimed to answer three key questions, each of which was addressed by a separate but complementary component of the evaluation methodology:
- How well do citizen-led assessments measure learning?
Methodology: Technical review of the testing tools, sampling design, and analytical processes used - How well do citizen-led assessment processes work?
Methodology: Process evaluation - How well do citizen-led assessments stimulate awareness and action about learning outcomes?
Methodology: Non-experimental evaluation of impact
What they measure, they measure well. Citizen-led assessments test a very constrained set of competencies in math and reading. The testing tools they use yield valid results, which cast a spotlight on limited achievement in these basic competencies. In this way they are important and useful, but broadening the testing tools and strengthening comparability would allow citizen-led assessments to do more to inform policy and practice. A selection of the evaluation’s key findings are provided below:
- Evidence suggests volunteers are well-equipped to reliably assess children’s basic competencies. Inter-rater reliability studies of ASER and Uwezo indicate a high level of agreement in volunteers’ scoring of children’s responses.
- At the international level, both ASER and, more recently, Uwezo, have contributed to the critical focus on learning outcomes in global discourse and agenda-setting. Their contribution has included both providing evidence of the seriousness of the learning crisis (i.e., revealing major deficiencies in even the most basic competencies) and demonstrating how a low-resource model can be used to assess learning on a national scale.
- Increasing awareness of the learning crisis at the national level is one of the main successes of both ASER and Uwezo. But generating concrete action to improve learning outcomes on the part of key stakeholders has proven much more challenging for both initiatives.
- For both ASER and Uwezo, only sporadic evidence of impact at the district level was found. This can largely be attributed to the lack of resources available for systematic involvement of the network of district partner organizations in dissemination activities, and, relatedly, to the limited capacity of these organizations.
- One small-scale but significant way in which both ASER and Uwezo have triggered learning-focused action outside of the government is through the uptake of the testing tools in education programs run by NGOs and CSOs.
- The evaluation uncovered only limited anecdotal evidence that participation in the survey stimulates awareness or action at the community level.
The work was undertaken with funding from the William and Flora Hewlett Foundation and with the help of a number of key partners, including the Australian Council for Educational Research (ACER), Catalyst Management Services (CMS) in India, OWN & Associates Ltd. in East Africa, Mr. Abdoulaye Bagayogo in Mali, Mr. Souleymane Barry in Senegal, and Qdata Enterprises in Kenya.
Findings from the evaluation can be found in the full report, available here.
