Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Benin: Malaria Control Booster Program


  
1. Project Data:   
ICR Review Date Posted:
09/24/2012   
Country:
Benin
PROJ ID:
P096482
Appraisal
Actual
Project Name:
Benin: Malaria Control Booster Program
Project Costs(US $M)
 31.0  33.0
L/C Number:
CH229
Loan/Credit (US $M)
 31.0  31.0
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
 0  0
Cofinanciers:
Board Approval Date
  06/01/2006
 
 
Closing Date
02/28/2011 06/30/2011
Sector(s):
Health (60%), Other social services (25%), Central government administration (15%)
Theme(s):
Malaria (40% - P) Population and reproductive health (20% - S) Health system performance (20% - S) Child health (20% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Judith Hahn Gaubatz
Judyth L. Twigg Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
According to the Project Appraisal Document (PAD, page 6), the objective of the project was:

  • to support implementation of the national 2006-2010 Malaria Control Program, the success of which will be reflected in the overall increase in access to, and utilization of, an effective package of malaria control interventions to reduce morbidity and mortality from malaria.

The project would contribute to strengthening malaria control capacities to:
  • improve case management and treatment of malaria so that by year 2010, at least 60% of malaria cases of children under 5 years old are treated adequately within 24 hours from onset of fever;
  • expand prevention activities so that by 2010 at least 60% of children under 5 years old and pregnant women sleep under insecticide treated nets (ITNs) and at least 60% of pregnant women receive a prophylactic treatment.

According to the Development Grant Agreement (DGA, page 13), the objective of the project was to contribute to strengthen malaria control capacities to improve case management and treatment of malaria, and expand prevention activities.

As the PAD statement of objectives makes reference to actual outcomes to be achieved - increased access and utilization - it is used as the basis for this review.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? No

Date of Board Approval:

c. Components:

1. Improving Case Management and Access to Treatment (Appraisal: US$14.7 million; Actual: US$8.1 million): This component aimed to improve malaria treatment capacity in the public health sector and among pre-selected NGOs. Activities included: updating of care protocols for Artemisinin-based combination therapy (ACT) and Intermittent preventive treatment for pregnant women ( IPTp); provision of drugs and basic equipment; and updating of case management methods. The project supported the government's new policy to phase out the use of chloroquine as treatment, and phase in the use of ACT.

2. Scaling Up Prevention Activities (Appraisal: US$13.0 million; Actual: US$21.3 million): This component aimed to implement mass community interventions, targeted to pregnant women and children under 5 years old. Activities included: large-scale distribution of long-lasting insecticide nets (LLINs) to target populations; introduction of IPTp for pregnant women attending prenatal visits; information and education campaigns (IEC); and training in the use of LLINs.

3. Strengthening Monitoring and Evaluation (Appraisal: US$1.7 million; Actual: US$0.9 million): This component aimed to strengthen malaria M&E and data management for the national program, at both central and operational levels, in coordination with key partners. Activities included: M&E systems development; expansion of sentinel sites; training; and conducting of studies and assessments.

4. Strengthening Program Management, Capacity Development and Promoting Regional Cooperation (Appraisal: US$1.6 million; Actual: US$2.7 million): This component aimed to support project management operations of the National Malaria Control Unit within the Ministry of Health (Programme National de Lutte contre le Paludisme - PNLP), as well as to promote regional cooperation. Activities included: training; policy development; operational research; establishment of a regional learning agenda; and implementation of a vector management plan.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost:

  • The actual project cost was US$33.0 million, compared to the appraised cost of US$31.0 million (the difference is attributed to fluctuations in the SDR/US$ exchange rate).

Financing:
  • The project was financed 100% by IDA.
  • During a project restructuring in 2010, an undisbursed amount of US$9.7 million was re-allocated to increase distribution of LLINs, increase stocks of treatment drugs, and conduct end-of-project surveys (hence adjusting the amounts for the relevant components). Allocations across disbursement categories were also adjusted due to higher-than-planned spending on rehabilitation, consultants, and operating costs.

Borrower contribution:
  • There was no planned borrower contribution; however, during negotiations, the government committed to allocate US$1.9 million annually to the PNLP for program costs. This commitment was met throughout the project period.

Dates:
  • The closing date was extended from February 2011 to June 2011, to allow completion of project activities, including the mass distribution of LLINs, surveys to document end-of-project results, research on the use of LLINs and ACT, and an audit of project accounts.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High. Given the high rates of mortality and morbidity among both children and adults due to malaria, the objective to increase access to and utilization of malaria interventions was highly relevant. The government's Poverty Reduction Strategy Paper (2011) and the Bank's Country Assistance Strategy (FY2009-2012) identify increased access to health services and treatments, specifically for malaria, as key outcomes. The Bank's Global Strategy and Booster Program for malaria also focuses on scaling up interventions, specifically increasing utilization of insecticide-treated nets (ITNs) and IPTp. Corporate priorities as reflected in the Millennium Development Goals include combating malaria, as well as child and maternal mortality.

b. Relevance of Design:
High. The project design supported both prevention and treatment interventions that were based on technical recommendations of the World Health Organization (WHO). Activities targeted populations (children and pregnant women) at greatest risk and were reasonably expected to achieve intermediate outcomes of increased access and utilization within the time frame of the project. Overall, the ICR (page 14) notes that the results framework of the project was highly consistent with Roll Back Malaria (RBM) and WHO logframes for malaria. The emphasis on strengthening M&E systems was likely to encourage the results-oriented focus.


4. Achievement of Objectives (Efficacy) :

To increase access to, and utilization of, an effective package of malaria control interventions to reduce morbidity and mortality from malaria. Substantial, due to evidence of increased access to, and utilization of, malaria interventions, alongside reductions in mortality (although these are only partially attributable to the project's support). While donor activities contributed to the achievement of program outcomes, the Bank played a significant role in implementation of activities and coordination of donor efforts. (The Bank contributed 65% of all available financing at the time of project approval, although the Global Fund provided a significant amount of funding in the ensuing years: $3 million in 2003-2006 for malaria, $28.9 million in 2004-2009 for ITNs, ACT drugs, and BCC, $14.5 million in 2008-2013 for ACT for children and health worker training, according to the Global Fund website.)

Outputs
Prevention
  • Provision of 3.1 million LLINs, exceeding the target of 2.2 million. Other donors (USAID, UNICEF) provided and additional 1.6 million LLINs.
  • Conducting of two mass LLIN distribution campaigns (in 2007 and 2011), including training of distributors, transport of LLINs to villages, and IEC activities. All 3,737 villages of Benin received LLINs.
  • Facility-based distribution of LLINs to target populations, specifically to children under five years at the time of measles vaccination and to pregnant women at the time of antenatal care (ANC) visits.
  • Contracting with 33 NGOs to increase knowledge about malaria transmission and to encourage prevention behavior. 31 of these contracts were renewed based on satisfactory performance.
  • Conducting of National Malaria Day events in 2008, 2009, and 2010.
  • Training of almost 2,000 health workers in IPTp policy and implementation.
  • Provision of over 593,000 doses of IPTp drugs (74% of planned amount). According to a project survey, 90% of health facilities reported no stockouts of more than one week during the previous three months. Other donors (PMI) provided an additional 1.1 million doses.

Treatment
  • Training of 10,000 health workers in improved case management, proper and routine diagnosis, and correct utilization of ACTs. The training was followed up by semi-annual supervision visits by central PNLP to the district level. However, supervision of more decentralized levels (health zones and community centers) was carried out less regularly than planned. Other donors (Global Fund) provided training for community malaria workers in treating malaria cases at the community level.
  • Distribution of 25.3 million ACT treatment kits to all public health facilities, including over 64,000 "A" kits and 49,000 "B" kits for the management of severe malaria cases. Other donors (USAID, Global Fund, and UNICEF) also financed an additional ACT treatments.
  • Provision of rapid diagnostic tests (RDTs) for all health facilities and microscope/laboratory supplies to 9 health facilities.
  • According to a project survey, 88% of health facilities reported no stockouts of ACTs of more than one week during the previous three months, and 79% of health facilities reported no stockouts of RDTs of more than one week during the previous three months. 33% of health centers had stockouts of ACTs for at least 3 days. Other donors (USAID) provided capacity building support on supply chain management.

Institutional capacity, particularly M&E
The ICR (page 20) notes that most studies, program evaluation surveys, and M&E capacity building efforts were coordinated and/or undertaken jointly by partners.
  • Establishment of a national M&E plan, with the inclusion of internationally-recognized indicators. The ICR (page 20) notes that the M&E framework is considered "a major achievement by the PNLP and serves as a reference for all partners supporting malaria efforts."
  • Conducting of a national mid-term evaluation in 2008, a Lot Quality Assurance Sampling Method (LQAS) in 2008, and a Malaria Indicators Survey (MIS) in 2010.
  • Establishment of a routine malaria information system, with collection of information on process indicators and results indicators.
  • Evaluation of the mass distribution campaign and conducting of other operational studies (efficacy of malaria drugs, vectors' resistance to insecticides, establishment of a pharmacovigilance system).
  • Training and provision of tools to health facilities to establish the information system for logistics management.
  • Development of prioritized, costed annual work plans based on joint reviews of performance and M&E data, culminating in "good coordination among partners, good complementarity of efforts, and... more of a seamless effort to ensure the reliable provision of program inputs and supports as donors come and go and evolve their support." (ICR, page 20).

Outcomes
Prevention
  • The proportion of households with at least one LLIN/ITN increased from 24.5% in 2006 to 74% in 2011, achieving the target of 70%.
  • The proportion of children under 5 years who reported sleeping under an LLIN/ITN the preceding night increased from 20% in 2006 to 64% in 2010, achieving the target of 60%.
  • The proportion of pregnant women who reported sleeping under an LLIN/ITN the preceding night increased from 20% in 2006 to 60% in 2010, achieving the target of 60%.
  • The proportion of pregnant women who received two or more doses of IPTp drugs during pregnancy increased from 3% in 2006 to 47% in 2011, nearly achieving the target of 50%. There were regional differences, however, with the highest district level reported at 61% and the lowest at 31%.

Treatment
  • The proportion of children under 5 years old with fever receiving treatment within 24 hours from onset of symptoms increased from 25% in 2006 to 47% at the community level and 95% at the health center level in 2011, surpassing the target of 30%.
  • A 2009 health facility survey raised some issues of quality, including: Less than 100% adherence to national policy on prescription of anti-malarials and malaria testing; Less than 41% of health centers able to conduct tests by microscopy or rapid diagnostics tests (100% of hospitals were able to conduct tests); "somewhat ambiguous" policy and training on testing, especially for children under five.

Malaria incidence and mortality
  • The annual number of simple malaria cases in children under 5 years old reported by health facilities increased from 365,852 in 2006 to 482,669 in 2009. The ICR (page 22) suggests that the project's information and communication activities encouraged the population to seek early treatment of malaria symptoms, thereby increasing the number of simple malaria cases reported while decreasing the number of cases that became severe (see next outcome). It is also plausible that the project's activities to improve case management and treatment (specifically provision of essential commodities and training for nationwide scale-up) contributed to an increase in the number of cases reported.
  • The annual number of severe malaria cases in children under 5 years old reported by health facilities decreased from 108,576 in 2006 to 76,493 in 2009. These support the observation that increased, early treatment of malaria cases has reduced the number of severe cases.
  • The annual number of reported malaria deaths among children under 5 years old decreased from 1,133 in 2006 to 767 in 2009.
  • The malaria case fatality rate among children under 5 years old decreased from 24/10,000 cases in 2006 to 14/10,000 in 2009.

The ICR (page 22) includes a discussion of malaria case reporting and fatality trends in the time period just prior to project implementation, in order to address attribution issues. While there was already a declining trend in the number of malaria deaths and case fatality rates, it is plausible that the continued decline during the project period is partially attributable to the project's support. The project comprised approximately 65% of all available financing at the time of project approval, thus representing a substantial proportion of the resources needed for nationwide scale-up. In addition, the infusion of resources and the improvement in PNLP's performance were critical in attracting additional resources for the program needed to further realize outcomes. There was no information provided in the ICR on rainfall patterns during the project period which could impact the number of malaria cases.

5. Efficiency:

Substantial. The PAD (Annex 9) includes a discussion of the economic aspects of the project, most notably the identification of high impact interventions (leading to reductions in child or maternal mortality) and analysis of bottlenecks in the delivery system.

The ICR discusses several aspects of the project that indicate substantial efficiency in the use of resources, although there is no project-specific information on cost-effectiveness. The project supported the most cost-effective interventions for malaria control, as identified by WHO. The high level of donor coordination "permitted a more effective and efficient use of their technical and financial support" (ICR, page 23). The malaria information system was integrated into the MOH's sector-wide system. Lastly, reliance on the health system to carry out malaria interventions avoided the creation of parallel efforts and structures.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The overall outcome of the project is rated Satisfactory, based on High relevance of the project objectives and design, Substantial achievement of the objective to increase access to and utilization of malaria control interventions, and Substantial efficiency in the use of project resources.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:

There remains strong commitment to malaria control programs, from stakeholders as well as from high levels of government. The key technical aspects of the project - the promotion of LLINs and ACTs - were incorporated into national policy and strategy. Institutional capacity in the PNLP was strengthened. Donor partners remain active and further Bank financing is anticipated through an ongoing Bank health operation in the country and a follow-up malaria booster operation in the region; however, there is some concern about the declining national budget for the health sector, which is needed to finance essential operating costs for program management.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:
The project design was based on extensive country-specific analysis, as well as the regional knowledge base. Interventions were based on technically sound recommendations of the WHO and incorporated principles of the Africa regional Booster Program, namely scaling up for impact and active collaboration with other donors. The institutional arrangements were clearly defined, adequately informed by assessments of fiduciary and overall institutional capacity. Key documents such as terms of reference for fiduciary staff, procurement plans, and draft contract agreements with NGOs were identified as conditions to be met prior to effectiveness. The critical role of M&E was also reflected in the legal covenant, calling for joint annual malaria reviews with donor partners and stakeholders. While the M&E framework was overall well-designed (See Section 10), there were discrepancies between the targets identified in the PAD and those in the DGA (Schedule 6).

Quality-at-Entry Rating: Satisfactory

b. Quality of supervision:
Supervision was highly proactive and focused on achieving results, reflected in part by the inclusion of M&E experts and WHO staff on supervision missions. The ICR (page 27) reports that "Aide-memoires and internal reporting kept the development outcomes in the forefront, and issues raised about delays or weaknesses in implementation were always addressed in light of their impact on achieving development outcomes." The highly satisfactory performance of the task team leader was consistently reported by stakeholders, citing: close, pedagogical supervision; candor and problem-solving orientation; rigorous assessment according to high standards; and a respectful, collaborative approach. Key project targets were revised in April 2010 (at which point the project had disbursed 70%) in light of updated baseline data and progress achieved at that point (See Section 10). Safeguard and financial management performance were overall satisfactory.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The government sustained its strong commitment to malaria control, as reflected by the adoption of a multi-year strategy, public statements about the prioritization of malaria control, and consistent provision of government budget for the malaria program. The government chose to use the remaining funds under its IDA allocation for the malaria project, which also signaled the high level of government commitment. However, procurement of essential commodities (drugs) experienced significant delays, affecting the launch of key project activities. The ICR (page 29) cites the National Procurement Agency and the Ministry of Economy and Finance as having taken months instead of the 15-day standard time period to process procurements and contracts (See Section 11).

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:
The primary implementing agency, the PNLP, performed satisfactorily. The utilization of PNLP staff to manage the project, rather than establishing a separate project implementation unit, helped to strengthen local capacity. PNLP maintained a strong focus on key project elements such as community involvement, M&E, and partner coordination. Review and planning meetings were conducted at regular intervals, even more frequently than planned. Although there were some initial problems in financial management, these were addressed and overall performance was adequate (see Section 11).

Other implementing agencies - NGOs (with the exception of two NGOs that did not perform satisfactorily and did not have their contracts renewed), the central medical drug agency, and research institutions were all reported to have fulfilled their implementation responsibilities.

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The M&E design reflected the critical need to strengthen monitoring systems, including broadening the scope to monitor behavioral as well as epidemiological trends, increasing the focus on results, integrating malaria indicators into the sector-wide M&E system, and serving donor partners as well as local health managers. The recognition of this need was reflected in the decision to dedicate an entire project component to M&E activities. Baseline figures were estimated according to 2004 data, but were subsequently updated according to 2006 data. Indicators were well-defined, including for target populations.

b. M&E Implementation:
The M&E system was mostly implemented as planned. Targets were revised in April 2010, with three targets on LLIN access and use amended to be more ambitious, and one target on treatment revised to a more modest level (although, as the ICR notes, the original baseline for this indicator included the use of chloroquine, while the target value excluded its use for treatment). The planned studies and surveys, including a Lot Quality Assurance Sampling survey, Malaria Indicators Survey, evaluation of the mass distribution campaign, studies on malaria drugs, vectors' resistance to insecticides, and establishment of a pharmacovigilance system, were carried out. Regional exchanges to share lessons learned from other country programs and international good practices were not implemented, as there were already other opportunities for regional learning.

a. M&E Utilization:
M&E data and joint reviews of performance contributed to decisions regarding program directions, including costed annual work plans of implementing agencies. Other examples of the use of M&E results include: targeting of health zones for corrective action based on LQAS results; renewing of performance-based contracts with districts and health zones based on indicators; choice of insecticide for bednets based on the evaluation of vector resistance; and the revision of targets based on updated data (2006 DHS and 2008 LQAS).

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:
The project was identified as an Environmental Category "B" project, triggering OP/BP/GP 4.01 on Environmental Assessment and OP 4.09 on Pest Management, due to the large amount of pests and pesticides to be handled. Although there were some delays in compliance due to low institutional capacity and weak cooperation among institutions, the medical waste and vector management plans were reported in the ICR (page 10) to have been implemented in a satisfactory manner.

b. Fiduciary Compliance:
Procurement: The National Procurement Agency was slow to review and clear procurements, while the Ministry of Economy and Finance was also slow to approve contracts, both of which led to delays in obtaining critical project inputs. There was some confusion about the delineation of responsibilities among the MOH procurement unit, the PNLP procurement unit, and the central medical drug store. Procurement performance substantially improved due to the arrival of a new project coordinator in 2008 and the implementation of strengthening measures.

Financial management: Although financial management arrangements were put in place early in the project period, there was subsequent weak performance due to underutilization of accounting software, failure to update accounting entries resulting in substantial delays in the submission of financial monitoring reports, and weak control of operating expenditures (as evidenced by anomalies in travel and training expenses). However, active intervention by the Bank team and timely response of the PNLP led to substantial improvements. There were no overdue audits, qualified issues, or accountability issues raised in audit reports.

c. Unintended Impacts (positive or negative):

d. Other:
NGO contracting: The initial NGO contracting process was undertaken at the central level, which led to weak ownership by health zones of the NGO activities, as well as weak accountability of NGOs to the health zones. However, the subsequent NGO contracting process (in the second round) was undertaken with the full involvement of the local level, with NGO activity reports now subject to the review of health zones. Selection criteria were also revised to be more rigorous and responsive to need, and therefore a few low-performing NGOs in the first round did not have their contracts renewed for the second round.



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Satisfactory
Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Exemplary
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

Lessons from the ICR (page 29):
  • Active, rigorous supervision of a project, particularly by way of a dedicated, field-based task team leader, can contribute to building the country's capacity to implement project activities. The guidance of the task team leader was consistently cited as being a critical success factor for this project.
  • Strong collaboration among government and partners can achieve significant synergies and efficiencies in planning, financing, implementation, and evaluation. Joint studies and evaluations, the second mass LLIN campaign, and the provision of essential commodities are examples of such collaboration from this project.
  • Addressing a single disease through a health systems approach, while leading to some inefficiencies and weaknesses, can be a vehicle for health systems strengthening. The project-supported malaria program, with its clear set of inputs, activities, outputs and outcomes, contributed to improved capacity in the health sector.

Lessons drawn from IEG:
  • A dedicated project component on M&E (with the dedicated resources) can highlight the importance of monitoring and ensure focus on results. In the case of this project, a full M&E component was designed to ensure sufficient attention and support.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR quality is rated Exemplary. The analysis of the achievement of outcomes is detailed and thorough, with a clear results chain of outputs to outcomes laid out. The discussion of outcomes also includes considerations of attribution (vis-a-vis donor contributions, although not rainfall patterns), regional variations within nationwide outcomes, and year-by-year trends within the entire project period. Relevant lessons from the project experience are well formulated. An ex-post cost effectiveness analysis would have further strengthened the outcome analysis.

a. Quality of ICR Rating: Exemplary

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions