Step 2: Data collection
Note: Data collection (Step 2) and analysis (Step 3) appear to be separate steps. But in reality, your first round of data analysis will and should send you back to do further data collection, especially talking to key informants, and then revising and deepening your data analysis.
By the end of Step 2, you should have:
- Finalized the adaptation of the Data Collection Tool (Word file{s}) and Data Synthesis and Analysis Tool (Excel file) to reflect the country terminology and scope of your application, based on your review of documents during this step.
- All the raw data collected and organized by the questions in the Word file.
- An initial synthesis/summary of the data in the cells of the Excel file based on first round of Data Analysis (Step 3).
This guidance on data collection will help you to:
- Be efficient with your time and the time of key informants
- Use your external reviewer(s) effectively to guide the data collection
- Build trust with stakeholders to speak openly
- Prepare for Step 3 and conduct an analysis that is accurate, insightful and useful
2.1 Cycles of secondary data collection and review
First you will prepare the Data Collection Tool (Word file) by editing the questions to match any changes and adaptations you made to the questions in the Data Synthesis and Analysis Tool (Excel file). Then you will make a separate copy of the revised Word file for each health financing scheme included in the assessment. While the Data Synthesis and Analysis Tool has all the financing schemes side by side, in this Data Collection Tool, you will use one document for each scheme and deep dive and record all the details, including references to your sources.
Table 2: The Data Collection Tool allows you to collect three types of data: normative, actual and subjective.
| 3 Types of data | Examples |
|---|---|
| Normative How is purchasing designed to function according to its policies, law, regulation, annual plans? How does the design compare with the Benchmarks? | A) A benefit package was defined based on population health data when the purchaser was established 5 years ago. B) The purchaser has a policy of selective contracting according to explicit standards. |
| Actual How is purchasing functioning in practice per objective data? Note deviations from normative and from the Benchmarks. | A) The benefit package is well specified but has not been revised in 5 years. B) In practice any provider can participate. |
| Subjective What are stakeholders’ perceptions about actual practices and performance compared to how purchasing is expected to function (normative)? What do they perceive to be the reasons why there are deviations? Why is actual performance better/worse than expected? What solutions have been discussed or tried? | A) The process to revise the benefit package is not clear to stakeholders. B) The purchaser felt pressure to add more providers, especially in remote areas, and the ability to verify providers’ compliance with the standards is limited. |
Get advice on the data collection process from experts who have applied the Framework:
Data collection and review: Round 1
In Step 1 you organized local stakeholders to support the application of the Framework. In Step 2, you work with members of your advisory group or technical working group and your MoH Champion to engage and prepare the purchasing agencies and others (e.g. regulatory agencies, provider representatives, consumer representatives) to contribute to the exercise by facilitating data collection and analysis. Early in the process, a general communication should be sent to the leadership of all the purchasing agencies to introduce them to the study objectives and process, to you and your team, and let them know that you will be requesting documents and key informant interviews. In addition, you may organize an event to kick-off the study. Make sure each purchasing agency designates someone as your point of contact for the assessment.
Before asking any of the purchasers for their internal data and documents, we recommend you begin by collecting, reviewing, and analyzing readily available published and grey literature. See list below in the Data Sources Table. Try to address as many questions as possible from these sources.
Technical review
You may choose to submit your first draft of data to your Technical Reviewer so he/she can suggest follow up questions and guide you how to dig deeper. This process allows you to identify information gaps, contradictions, and potential issues so you can target your requests for internal documents and target your questions in key informant interviews. This approach is more efficient and builds your credibility with stakeholders and key informants.
At this stage, you may also review the number of schemes that have been selected to be included in the assessment. You may have discovered there was an important scheme missing that needs to be included, or too many have been selected and may need to be rationalized to allow for more in-depth data collection and analysis. You may also find that some schemes are very similar in structure and implementation and may need to be harmonized. For example, in many low- and low-middle-income countries, there are vertical programs for delivery of priority services such as HIV, malaria, tuberculosis and immunization that have their own financing, planning and service delivery structures. In most settings, although these programs are implemented in parallel within different agencies, their purchasing function may be quite similar and so one scheme is selected for detailed assessment as a proxy for the others.
PRO TIP: Two technical reviewers were assigned to review all the data collected across the ten African countries. Written comments were provided, and meetings were scheduled with each research team to clarify comments or questions and plan the way forward for round 2 of data collection and key informant interviews.
Data collection and review: Round 2
Once you have completed collecting, reviewing, analyzing and extracting data from publicly available documents (Round 1), you are ready to contact the purchasing agencies for further data collection. We recommend you begin by meeting with your point of contact for each purchasing agency to build rapport and plan next steps. Use the meeting to review the objectives and scope of the study, provide an update on your progress, and encourage your contact to ask questions. Together, you need to clarify the process for contacting staff within the purchasing agency to request documents and interviews. Be prepared to assume responsibility to prepare multiple emails tailored to each request that:
- Introduces you and the study by briefly explaining the objectives and process of the study. Even if there have been previous encounters, repeating this information helps the recipient explain the study to his/her colleagues and respond to your request.
- Specify the topic(s) you need to address, the documents you seek, and/or key informants you would like to interview.
2.2 Data sources
The table below presents examples of documents and key informants for each section of the Strategic Health Purchasing Progress Tracking Framework.
See Table 3: Examples of documents and key informants
| Data Sources | ||
|---|---|---|
| Topic | Documents | Informants |
| Governance and institutional arrangements of health financing schemes and purchasing agencies | - Government/MoH publications: national policies, plans and strategies for health sector, health financing strategies and assessments, health insurance reports and audits - Organograms, terms of reference, role and responsibilities of relevant MoH units, purchasing agency, relevant oversight and regulatory bodies - Purchasing agency authorizing legislation, charter, regulations, operating guidelines - Plans and Manuals for health financing schemes targeting specific populations, services, or geographic areas such as Results Based Financing (RBF), MCH, rural communities etc. | Senior staff in relevant MoH units, purchasing agencies Members of relevant regulatory agency Legislative committee for health sector, health financing |
| Expenditure Management | - National Health Accounts - WHO Global Health Expenditure Database - Country reports - Government/MoH publications, websites on public health financing, health budget and spending - Public expenditure tracking surveys that include health sector (World Bank) - UNICEF tracking of off-budget health financing - Purchasing agency financial performance data - Financial performance data for specific health financing schemes such as Results-Based Financing, MCH | Purchasing agency’s senior finance staff Ministry of Finance unit for health sector financing |
| Public Financial Management | - Government/MoH PFM legislation, manuals, guidelines especially: - specific to the health sector - health budget allocation criteria and procedure - procurement and contracting - Public expenditure tracking surveys (World Bank) - Government internal audit agency reports e.g. Office of the Auditor General annual reports | MoH senior financing staff Ministry of Finance units for decentralization, health sector financing |
| Service Readiness | - Government/MoH health information system data on service access and utilization - WHO Service Availability and Readiness Assessment (SARA) - Population surveys on health service utilization Demographic Health Surveys, other? - Service Provision Assessment (SPA) | MoH leadership for health services Development partners involved in delivery of health services |
| PF1. Benefit specification | - Government/MoH declaration of essential health services, minimum healthcare package, scopes of care by level (primary, secondary, tertiary) - Purchasing agency publication of services covered (explicit list), exclusions, process for updating the benefit package - Health insurance regulatory agency – required minimum packages, process for updating requirements - Plans and Manuals for health financing schemes targeting specific populations, services, or geographic areas such as RBF, MCH, rural communities etc. | If it exists, members of the body responsible for benefit specification and revision. Could be within MoH or independent. Purchasing agency staff Regulatory agency staff |
| PF2. Contracting arrangements | - Government/MoH PFM legislation, manuals, guidelines for procurement and contracting; sample provider contract - Plans and Manuals and sample provider contracts for health financing schemes targeting specific populations, services, or geographic areas such as Results-Based Financing, MCH, rural communities etc. - Purchasing agency policies, manual, guidelines on provider contracting; copy of a contract - Norms and standards for facility infrastructure and staffing | Purchasing agency staff responsible for enrolling and contracting providers Provider groups |
| PF3. Provider payment | - Government/MoH PFM legislation, manuals, guidelines for provider payment - Plans, Manuals, guidelines for paying providers for health financing schemes such as RBF, MCH, rural - Purchasing agency policies, manual, guidelines on provider contracting; copy of a contract | Purchasing agency staff responsible for paying providers Provider groups |
| Performance monitoring: Provider Level Purchaser Level | - Quality assurance guidelines - HMIS e.g. DHIS 2 - Supportive supervision guidelines - Provider contracts that describe what data providers must submit, and how the purchaser will monitor provider performance - Descriptions of the purchasers’ information technology and systems for managing providers and the quality of care provided - Reports from the purchasers’ monitoring unit that describe how data are shared and used by providers, and used by the purchasing agency for purchasing decisions (e.g., design/redesign payment methods) - Purchasing agency authorizing legislation, charter, regulations that describe how purchasing agency performance will be monitored, who monitors, how do they monitor, and what decisions and actions can they take - Descriptions of the purchasers’ information technology and systems for managing financial performance (claims ratio, expenditure ratio, renewal rate, budget, revenue, and expenses) | MoH leadership and subnational MoH staff responsible for health service quality and health information systems Provider groups Purchasing agency staff responsible for enrolling, contracting, and paying providers; and staff responsible for design/redesign of payment methods Regulatory agency or body responsible for oversight of purchasing agency’s performance Purchasing agency staff responsible for design/redesign of payment methods |
| Information technology | - E-health policies and strategies - Digital health legislation/regulation - Visit each purchasing agency’s website - There probably will be different websites for beneficiaries and providers | Chief Information Officer at each purchasing agency Provider groups and beneficiaries feedback on their experience with purchaser’s IT |
| Communication with beneficiaries (see below) and providers | - Communication strategies, policies for each scheme’s beneficiaries and contracted providers - Any evaluations or data on how well schemes communicate with beneficiaries and contracted providers - Review scheme websites that have different portals (log in page) or separate websites – one for beneficiaries and one for providers | Titles and organizational structure will vary: Manager of communications, beneficiary or membership relations, provider contracting/credentialing |
The 2015 final report of the evaluation of Community Based Health Insurance pilot schemes by the Ethiopian Health Insurance Services (EHIS) showed that knowledge about the scheme was 95% for both members and non-members, which was the highest and attained through the dissemination of information through informed neighbors, CBHI officials, or house-to-house sensitization; that also seemed to be an effective means to improve the beneficiaries’ satisfaction with the scheme.
Example of data sources on communication with beneficiaries
2.3 Key informant interviews
The purposes of key informant interviews include validating information and your understanding and getting informants’ perceptions about how and why actual practices and performance deviate from how purchasing is expected to function according to official policy/regulations (normative) and the Benchmarks. What do they perceive to be the reasons why there are deviations? What solutions have been discussed or tried?
To identify key informants, talk to your point of contact for each purchasing agency, the stakeholder advisory group, technical working group, and/or the MoH champion. They can help you identify the right people and make introductions, including authorization for them to speak with you if needed. As mentioned above, be prepared to send individual emails to request a key informant interview.
Remember to:
- Introduce yourself and the study by briefly explaining the objectives and process of the study. Even if there have been previous encounters, repeating this information helps the recipient respond to your request.
- Present official authorization/permission to request information (if needed).
- Reassure that this study is not an audit or evaluation, but a learning effort.
- Confirm that his/her responses are anonymous.
- Specify the topic(s) you would like to address with this particular interview, possibly including specific questions.
PRO TIP: In Tanzania, the research team input all the data collected and, with support from the technical reviewer, identified the questions where there were gaps in information that needed clarification, and the agency best placed to respond to these questions. This allowed the research team to target selection of key informants from each agency, the institution and role/job function, and prepare specific interview tools based on the gaps in secondary data.
2.4 Data collection for results analysis
In Worksheet 5) Results Analysis, we aim to understand and explicitly draw out whether:
- The purchaser is using the purchasing functions as levers to improve resource allocation, incentives to providers and accountability.
- The effects of purchasing on intermediate and final UHC objectives.
Results analysis requires a review of broader evidence at the system level to compare with the scheme-level data.
- See Table 4 below for useful data to collect for your results analysis. For example: reports on scheme performance, strategy documents, policy documents and program reports from the Ministry of Health and other regulatory or policy institutions, reports based on routine data collection, population surveys such as the DHS. These documents may be published or internal.
- For results analysis, it is recommended to also conduct a web search for peer-reviewed articles using a combination of key words including: “country name”, “strategic health purchasing,” “health financing,” “service delivery,” “outcomes,” “accountability,” “incentives for performance,” and “resource allocation”.
- Key informant interview questions focused on each health purchasing function and associated results: appropriate incentives, cost-effective resource allocation, accountability for quality, and UHC goals and objectives.
Indicators
Results can be measured by comparing indicators. Table 4 provides some examples of possible indicators for each question. Depending on the data available, indicators can be compared:
- Over time (trend data), before and after the introduction of the scheme or specific purchasing method and/or
- Geographically if the schemes operate in different geographic areas or were rolled out incrementally.
In all cases, the team must recognize the many other factors, such as social determinants of health, which affect some of these indicators.
See Table 4: Possible indicators for Result Analysis
| Results | Result Analysis Questions | Possible Indicators | |
|---|---|---|---|
| 4.a. Appropriate incentives | 4.a.(1) | To what extent do provider payment methods incentivize the delivery of high-value services (e.g., PHC) and to serve vulnerable populations? | - Make a matrix of a representative list of high value services and vulnerable populations tailored to the country (1 axis) and what each scheme covers - Service indicators should reflect the benefit packages - Immunization rates - Prenatal care from skilled provider - Modern contraception prevalence rate |
| 4.a.(2) | To what extent are provider payments harmonized or not harmonized across schemes/revenue sources to ensure coherent incentives for providers? | - Number of schemes and number of provider payment methods: Total /national and subnational; Total / type of provider - Ranking of schemes by amount of financing and number of lives covered |
|
| 4.a.(3) | Are there any adverse incentives in the system, leading to inefficiency or poor quality? | Qualitative data, narrative description | |
| 4.b. Cost-effective resource allocation | 4.b.(1) | Is there evidence that funding allocations and payment to providers reflect population health needs? | Comparison of data on health needs (mortality and morbidity data) with claims data or expenditure of related services e.g. maternal mortality and amount of resources used for maternal health and family planning services |
| 4.b.(2) | Has any progress been made ensuring funds are not concentrated in urban wealthy areas? | Socio-geographic distribution of beneficiaries and enrolled providers Trend over time in scheme expenditures by type of beneficiary and provider |
|
| 4.b.(3) | Have purchasing arrangements and provider payment systems encouraged an increase in the share of funds allocated to PHC? | Share of scheme resources channeled to PHC and other cost-effective services | |
| 4.c. Accountability for quality | 4.c.(1) | To what extent do provider payment methods and purchasing arrangements promote quality of care and coordination across levels of care? | - Provider payment mechanisms are designed to enhance gate-keeping role of PHC providers to avoid unnecessary hospitalizations - Rate of primary care-sensitive admissions |
| 4.c.(2) | To what extent are purchasing arrangements used to promote or encourage quality of care at the provider level? | Quality assurance systems exist and used regularly and effectively | |
| 4.c.(3) | How are providers held accountable to provide high-value services (e.g. PHC) and serve vulnerable populations? | - Share of resources flowing to PHC - Average PHC spending per beneficiary - Share of enrolled/ - Registered individuals - Seeking primary care |
