Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Health Systems Reconstruction


  
1. Project Data:   
ICR Review Date Posted:
02/11/2013   
Country:
Liberia
PROJ ID:
P105282
Appraisal
Actual
Project Name:
Health Systems Reconstruction
Project Costs(US $M)
 8.51  8.73
L/C Number:
CH304
Loan/Credit (US $M)
 8.51  8.73
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  06/14/2007
 
 
Closing Date
10/01/2010 10/01/2011
Sector(s):
Health (56%), Central government administration (28%), Tertiary education (16%)
Theme(s):
Health system performance (33% - P) Child health (33% - P) Population and reproductive health (17% - S) Conflict prevention and post-conflict reconstruction (17% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Elaine Wee-Ling Ooi
Judyth L. Twigg Navin Girishankar IEGPS2

2. Project Objectives and Components:

a. Objectives:
The project was an emergency operation. Its original objectives, as stated in the Financing Agreement (page 4), were "to assist the Recipient in its post war efforts to reconstruct its health sector, through: (a) strengthening of policy making and management functions at the Ministry of Health and Social Welfare (MOHSW); and (b) providing critical inputs to sustain the referral system needed to support essential health services."

The project development objective as stated in the PAD is slightly different: "In partnership with other donors the Project will contribute to: (a) strengthening of policy making and management functions at the Ministry of Health and Social Welfare (MOHSW); and (b) providing critical inputs to sustain the referral system needed to support essential health services. "

The project was restructured in January 2010. The revised objective (as articulated in the Amendment to the Development Financing Agreement) was to: "(i) strengthen the policy framework and selected management functions of the MOHSW; and (ii) improve pre-service training and selected components of the basic package of health services."

The objectives stated in the Financing Agreement and the PAD are slightly different, though not inconsistent. The Guidance Note (paragraph 4 Resolution of Differences) given to operational staff with respect to packages submitted to the Board notes that “If there are differences of views between the lawyer, other reviewers, and/or the Task Team Leader as to the consistency of the project documents with the negotiated legal agreements, the negotiated legal agreements prevail absent a reopening of negotiations to address the discrepancy. “ Therefore the IEG will take the objective as stated in the Financing Agreement to assess Efficacy before and since the restructuring in January 2010.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 02/07/2010

c. Components:

Component A: Support Systems (appraisal estimate: US$2.37 million; actual cost: US$2.48 million)

This component aimed to support the operational capacity of the MOHSW to effectively manage the project by, inter alia, strengthening its management capacity, and expanding its monitoring and evaluation mechanisms, through the provision of technical advisory services, training, and incremental operating costs.

Component B: Human Resources (appraisal estimate US$2.83 million; actual cost US$2.38 million)

This component was to support the hiring of clinical teachers (including medical doctors, nurses, midwives, and allied health workers) to train and supervise clinical care staff. It was to provide technical assistance and training to public education institutions to reform their curriculum to enhance the education level of medical doctors, nurses, and allied health workers. It also aimed to provide training, technical advisory services, and equipment to enhance the technical and managerial capacity of key staff in the Recipient's health sector including, inter alia: (a) transportation costs of trainers and trainees; (b) per-diem of trainers and trainees; (c) workshops; (d) seminars and courses; (e) rental of training facilities and equipment; (f) acquisition of training equipment and teaching material; (g) internet access to online learning; and (h) study tours.

Component C: Infrastructure and Equipment (appraisal estimate: US$3.31 million; actual cost: US$ 3.87 million)

This component was to support rehabilitation of essential medical facilities and medical teaching institutions, including the maternal and child wards at the John F. Kennedy Hospital and the A.M. Dogliotti College of Medicine (COM). In addition to this, it was also to support the acquisition of basic medical supplies, medical emergency equipment, radiology and laboratory equipment, communications equipment, energy generating equipment, internet connection equipment, and ambulances for MOHSW clinics and health centers, county hospitals, and the John F. Kennedy Hospital.

The restructuring in 2010 modified the scope of work, and the revised design focused on a more finite set of interventions. The few changes (described in the Action Plan, Annex 3 of the Revised Project Data Sheet) included:
i) dropping of “development of a contracting policy funded by USAID” from the project’s results framework;
ii) eliminating support to upgrade training curricula (to be supported by other donors) ;
ii) adding pre-service training of radiographers and nurse anesthesists; and
iv) adding support for medical waste management at a selection of facilities renovated by the project.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Costs and Financing:
This was a Bank emergency grant of US$8.51 million. There was no counterpart financing to the project nor were there cofinanciers. The grant amount remained the same throughout the life of the project.

Dates:
The project was formally restructured (1/28/2010) after the mid-term review (conducted on 2/13/2009) in order to: i) scale down the ambitious scope for a 3-year emergency project, ii) be less dependent on other donor actions, and iii) be more targeted due to modest finances available and rising commodity costs. At the Restructuring date, the project had 46% of the grant disbursed.

The project was extended by one year and was completed on 10/1/2011.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Relevance of objectives is Substantial.

The project's original and restructured objectives were substantially relevant, aimed at helping Liberia's Ministry of Health to strengthen its stewardship function and manage the transition from emergency/ humanitarian relief to development in health. Liberia was in the immediate post conflict stage where 80% of health services were directly provided by foreign non-governmental organizations (NGOs) and humanitarian relief organizations that would soon leave the country. At appraisal the project was consistent with the interim Poverty Reduction Strategy for 2006 and the Interim Strategy Note of 2008. Both strategies sought to secure equitable access by different segments of the population to an efficient and effective health system. At project close the original and revised project objectives remained relevant to Liberia’s Country Assistance Strategy for 2009-2012 under the third pillar of facilitating pro-poor growth. The objectives were relevant to three pillars (health financing, health manpower resources, and health support systems) of the National Health Policy and Plan (NHPP); other donors committed to the fourth pillar "Basic Package of Health Services". The project objective supported an improved policy environment during post conflict conditions.

b. Relevance of Design:
Relevance of design is Substantial.

The original project design was modest. It envisaged that the Bank would be working closely with other donors and there would be significant interdependence between the different donor supported activities. The design was too ambitious in scope. Some of the project activities in the original project design and results framework included those supported by other donors, including development and implementation of new procedures in contract management; and development of a national health management information system policy. There was no disconnect between planned activities and outcomes, but there was overstatement of the Bank's contribution. The project was revised after the mid-term review and the scope of work modified. The revised design was substantially improved. It provided clarity on linking activities to objectives, and results framework provided clearer attribution as activities supported by other donors were dropped.


4. Achievement of Objectives (Efficacy) :

Original objective: (a) strengthening of policy making and management functions at the Ministry of Health and Social Welfare (MOHSW); and (b) providing critical inputs to sustain the referral system needed to support essential health services.

Revised objective: "(i) strengthen the policy framework and selected management functions of the MOHSW; and (ii) improve pre-service training and selected components of the basic package of health services.

The results framework includes indicators that were affected by factors outside the scope of the project including support provided by other donors. As a result, it is difficult to draw a results chain directly from the project’s activities to its outcomes and impact measured by the project’s formal indicators.

The first part of the project's objectives are identical and assessed together:

(i) Strengthen policy making and management functions at the Ministry (original and revised): Substantial

Outputs:

The Bank supported operational capacity building at the Ministry of Health to effectively manage the project, and expand monitoring and evaluation.

  • Senior level staff (including 2 deputy ministers ) augmented their skills in leadership, project management, and financial management.
  • 8 procurement staff (target was 5) in the Ministry were trained in accordance with Liberia’s Public Procurement and Concessions Act. Two senior staff received Master's level training in Italy.
  • 28 other ministry staff received training in procurement, financial management, and health management information systems.
  • The Bank through this project collaborated closely with the British Department for International Development (DfID) to build the capacity of the Office of Financial Management (OFM) within the Ministry of Health, helping it establish sound accounting, internal auditing, reporting, and other financial management procedures and systems.
  • The Ministry conducted four (joint) reviews with donor partners of progress towards implementation of the National Health Policy, as well as action plans. Two such reviews were completed at restructuring, and four reviews were completed by project closing.
  • All 15 county health offices in Liberia were equipped through the project with a computer systems to produce quarterly reports, meeting the target. USAID financed the design and implementation of the Health Management Information System (HMIS).
  • Relevant staff were trained in information technology and given responsibilities to operate the system.
  • Special studies on Liberia’s health manpower resources, health financing, and decentralization of health services were conducted with USAID support. The USAID financed policies were drafted and submitted to the Liberian Cabinet for approval. These policies were expected to inform national debate on health sector reforms, and to help guide the allocation of resources within the sector.

Outcomes:
  • The Ministry has strengthened its evidence base on health policy. The Ministry uses findings from analytical work generated by different donors to guide health spending. Per capita spending in health was increased from US$ 19 in 2007 to US$ 33 in 2010 (ICR, page 17).
  • The Office of Financial Management at the Ministry of Health took over the financial management responsibility from the Ministry of Finance and has now fiduciary oversight of donor funds in health. OFM now financially oversees donor projects in health. Although not discussed in the ICR, IEG found through interviews for the Liberia Country Partnership Evaluation that this was successfully accomplished as per arrangements in the PAD (page 12). Due to the strengthened and trusted position of the Ministry, USAID (reported in IEG interviews for the Country Partnership Evaluation) has broken from customary practice and has allowed discretionary funding support to the Ministry. According to USAID, the Bank’s contribution (working closely with other donors via this project) helped strengthen a key area in the country's underlying health systems.
  • Procurement is handled directly by the Procurement Unit within the Ministry of Health, with some oversight by the Ministry of Finance.
  • HMIs data are now generated systematically and fed into quarterly medical and epidemiological reports. Counties have begun to use the information generated by the HMIS as a source to guide their decision making (Borrower’s ICR, page 39).

The second parts of the original and revised PDOs was slightly changed in 2010 and will be assessed individually:

(ii) Provide critical inputs to sustain the referral system needed to support essential health services (original 6/2007-1/2010), Modest

Outputs:
  • Seven key secondary referral hospitals were supplied with modern X rays and laboratory equipment.
  • One functional secondary referral center per county increased from 15% of counties at baseline to 27% in 2009 not meeting the mid-point target to 47%.
  • The 2 maternal and child health wards in the JFK hospital and in 15 referral centers were renovated and now functional, meeting the target, This expands the availability of maternal and child health services in the referral system.
  • 200 health clinics received medical and laboratory equipment, meeting the target.
  • 102 hospital beds were added to the system. The number of pediatric beds increased from 28 to 88.
  • 222 health facilities were constructed, renovated, upgraded, or medically equipped.
However, it seems unlikely that the project with $8.7 million was the only financing source to achieve these targets, rather, other donors must have co-financed some of these investments in equipment and infrastructure.

Outcomes:
  • The placement of staff and modernized equipment has enhanced diagnostic and case management capability in the seven referral hospitals (see Borrower Comments, ICR page 39). The ICR (page 15) notes that this has enabled facility-based deliveries, successful medical and surgical outcomes, and improved inpatient care in underserved areas.
  • The Borrower's ICR (page 39) asserts that the rehabilitated maternity and pediatric wards have begun to alleviate overcrowding, reduce cross infection, and improve patient care.
  • The Accreditation Report in 2010 for the Basic Package of Health Services found 80 percent of government facilities met minimum standards (laboratory, medical supplies and staffing) for delivering the Basic Package.
  • Compared to 2008 (Population Census Report 2008) when 40% of all households traveled one hour or more to the nearest health facility, the 2010 Core Welfare Indicators Questionnaire (CWIQ) Survey, showed that 86.8 percent (87.1% in rural areas and 86.4% in urban areas) of the population had access to a facility within 5 kms or one hour away (the MOHSW Standard). The CWIQ also found improved affordability of care. 89.6% of people in need of health care visited a health provider, and there was little urban-rural difference (89.5% rural, 89.7 urban).
  • However, the ICR does not provided specific data on referrals, occupancy rates, and infection rates. Various studies reported in the ICR show improved access country-wide.

(iii) improve pre-service training and selected components of the basic package of health services (revised 1/2010 - 10/2011): Substantial

Outputs:
  • The Tubman Institute of Medical Arts and the Dogliotti College of Medicine were training institutions for health and medical manpower. Both institutions were renovated and equipped with laboratory and diagnostic equipment, supplies, textbooks, and teaching aides.
  • At Dogliotti, 4 teaching labs were renovated but only 2 were fully functional at project closing.
  • The share of health facilities with improved equipment score increased to 66% exceeding the target of 50%.
  • Seventeen clinical professors and instructors were recruited (mostly contracted from overseas) to Tubman and Dogliotti to train new health workers. The dearth of skilled Liberians and difficulty in retaining overseas experts due to difficult working conditions in the country posed a substantial challenge to the project.
  • Training curriculum for nurses and midwives were revised and in use. This activity was supported by other donors.
  • The project supported the pre-service training of radiographers and nurse anesthesists. 41 radiographers and 12 nurse anesthesists were graduated by project closing. The graduates were placed in the seven key referral hospitals.
  • Approximately 900 doctors, nurses, nurse anesthetists, radiographers, and other essential health workers were trained at an accelerated pace and graduated from the two institutions.
  • About 600 of the graduates had received project-financed stipends.
  • Annual enrollment at both institutions reached 647 (exceeding the project target of 550). Borrower comments (ICR, page 39) note that “for the first time in years, students enrolled at the two training institutions were able to complete their studies on schedule, instead of having to defer graduation due to the lack of instructors and teaching equipment and materials."

Outcomes:
  • The country’s health manpower base saw a 10% increase in essential health workers (including 30 doctors).
  • The 600 graduates with stipends were assigned in areas of critical need and bound by two-year contracts with the government (ICR, page 16). Thus, even though manpower shortages persist, the project has contributed to increased availability of skilled staff at least for the near term.
  • The availability and coverage of the Basic Package of Health Services increased from 32% in 2008 to 84% in 2011. the Basic Package consists of a standard set of prevention, care, and treatment services that are affordable and cost effective, and provided at each level of the health system. The package varies depending on the level of care.

5. Efficiency:

The PAD and the ICR do not include an economic analysis. The PAD (pages 4-10) argued that the project would be good value for money because it chose to support the areas in the National Health Plan where there was no other donor financing (in areas of health financing, and in studies that would guide better allocation of the sector’s limited resources); however, neither the PAD nor the ICR provide the analysis to support this statement. The ICR was unable to calculate cost benefit analysis. It notes (page 5) that "limited technical analysis of either the cost effectiveness of the proposed intervention, or alternative approaches, and a poorly designed results framework" made calculating a cost benefit analysis difficult. No project IRR or ERR were calculated.


The overall efficiency of project implementation was compromised, by the very slow first two years of implementation when only 14.1% of the budget was disbursed. Project restructuring resulted in consolidation of physical targets and some scaling back. After restructuring, implementation arrangements were more streamlined, and procurement management received technical assistance and dedicated support. As a result, the project achieved almost all key activities and targets in the last year of the project.

Efficiency is rated Modest.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The project's objectives were substantially relevant to Liberia's post-conflict situation and to the Bank's country strategy. Design relevance was Substantial. The first objective was substantially achieved. The second objective before restructuring was modestly achieved (46% disbursed) and after restructuring substantially achieved (54%). The Bank-sponsored training and construction/rehabilitation in collaboration with the support by other donors, notably USAID and the Italian government, contributed to increased Ministry capacity for policy making, procurement, financial management and data management, and human resources and the referral system strengthened to the point that there was markedly increased availability and coverage of the Basic Package of Health Services. Efficiency is rated Modest in the absence of any economic analysis and initially slow project implementation.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

Liberia will still be heavily dependent on external aid. National Health Accounts show that sector financing has improved from $60 million in 2007 to $130 million in 2010. Per capita health spending has increased from US$ 19 in 2007/2008 to US$ 33 in 2009/2010. Donor financing, however, has gone up from 72% of sector financing in 2007/2008 to 83% in 2009/2010 (due in part to confidence in the sector and evidence of gains), raising sustainability and predictability concerns. Due to the high dependency on donor aid, a Significant risk rating is assigned.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

During project preparation, Liberia was transitioning from emergency relief to reconstruction. The Bank team was responsive to the urgency of support needed. The Bank quickly mobilized funds in March 2007 under OP/BP 8.0 (Rapid Response to Crises and Emergencies), despite the fact that Liberia’s arrears to the Bank and other Multilateral Development Banks had not yet been cleared. Bank support focused on the pillars of the National Health Policy and Plan not covered by other development partners. This ensured that all key pillars of Liberia’s health system reconstruction efforts were addressed. During project preparation, the Bank used its bargaining power and good relationships with donors, to ensure that donors contributed to a common agenda. The project was prepared in less than four months and became effective less than a month after Board presentation.

However, the ICR (p. 21) reports serious shortcomings in project preparation which slowed down implementation. Given the post conflict conditions and the fact that Liberia was a first-time client, the procurement-loaded project was too complex to be completed in three years. The project was also too ambitious in scope given the institutional, capacity and post-conflict context of the country at the time. By design it attempted to harmonize donor assistance (the health sector had a higher number of donors and international NGOs than any other sector) by having a common work program and shared objectives. However, IDA funds were inadequate to support planned activities, and the results framework was unclear including indicators and targets not supported by this project. The Bank could have done more to ensure the project was better designed. An improved design would have supported the preparedness of the client for project implementation (ICR p. 22).

Most government institutions were not fully functional and experienced acute shortages of trained professionals in all disciplines. This created severe capacity gaps in all sectors. The Bank did not assess capacity gaps and did not make adequate arrangements for capacity building in Bank procedures for a first time Client during the preparation phase. Shortcomings included insufficient assistance in preparation of the project operations manual, procurement plans, development of the required environment and social management framework, and medical supplies and medical waste management plans, as well as adequate budgetary provisions for these activities. Project activities involved small but multiple goods and services, and procurement requirements were unwieldy. Emergency operations were expected to benefit from more flexible procurement procedures, but this did not happen. The Bank should have ensured training in a weak capacity environment and insisted that the operational manual be completed. As a result of these shortcomings during project preparation, project implementation of this operation, - which was prepared under Emergency Procedures - was severely stalled during the first two years.

Quality-at-Entry Rating: Unsatisfactory

b. Quality of supervision:

Quarterly supervision was planned to jumpstart the project. In the early stages of supervision, problems were noted in the implementation status reports but were not adequately resolved. Design weaknesses and the extremely limited institutional and implementation capacity, which were not addressed by the Bank team during the preparation phase, demanded significant support and ‘hand-holding’ by the Bank team throughout the entire project implementation period. As a result, only 14.11% of the grant was disbursed at midterm review in February 2009 (ICR, page 11). After the mid-term review, the Bank in January 2010 restructured the project, changed the scope of work, and extended the project by one year from October 2010 to 2011. However, it took the Bank team a long time - one year - to process this restructuring and address the project weaknesses identified during the mid-term review.

The restructuring brought focus, relegated certain activities to other donors, and brought greater delineation between IDA and other donor support. The revised activities also gave immediate priority to boosting the Ministry’s capacity in procurement. Bank-supported activities were consolidated. Implementation arrangements were streamlined, and a project coordinating unit was set up in the Ministry to coordinate communications and activities between the different project beneficiaries (Ministry, tertiary medical schools and hospitals, etc.) and the Bank. The scaling back, consolidation of activities, improved implementation arrangements, and extension of project duration enabled the project to meet its overall goals and targets. Thereafter, technical assistance was stepped up and the Bank focused appropriately on procurement issues. When these were cleared up, implementation picked up quickly. Supervision missions were more regular but were still insufficient. The Bank budget essentially limited supervisions to a maximum of two per year, against the four originally were planned.

The Bank was heavily engaged in policy and strategic discussions with the client and other donors. Despite the initial slow progress in implementation, the Bank has leveraged the $8.5 million to close collaboration with other donors. USAID, Liberia’s largest and most important donor, valued the Bank's contribution and its comparative advantage in policy and in health sector financing (IEG, Liberia Country Partnership Evaluation, June 2012).

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The Government has been very supportive of the project. In hopes of securing equitable access to health services in the post conflict period, the Government abolished user fees at the primary level, while promising a comprehensive package of health services (BPHS, Basic Package of Health Services). The government has further committed to adding services for neglected tropical diseases, non communicable diseases, and school health etc into an expanded Essential Package of Health Services. This package was not adequately costed, however, and funds (both donor and government) are not sufficient to meet the full sector needs (drugs, laboratory work, staffing at facilities etc). Allocation for health from the Government has increased in absolute terms, from $11 million in 2007 to $33 million in 2010, but the share of Government spending in overall health spending has remained the same. The country remains very dependent on donor assistance. The draft National Health Accounts in 2009/2010 found the donor share of total institutional health financing had increased to 83% from 72% in 2007/2008. There are indications the government will reintroduce user fees on a limited or phased basis.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
The Ministry of Health was fully committed to the project, and the stewardship by the Minister of Health and senior deputies were instrumental. The strong reputation and overall performance of the Ministry of Health were the reasons the Ministry of Finance relinquished financial management oversight for health programs to the Health Ministry.

At the implementation level, however, and particularly prior to the restructuring, there were coordination problems. A Project Coordinator's office had been created as the focal point for all the different implementation agencies (technical departments of the Ministry; Planning, Research, and Development; Health Services; and the two hospitals) in the project, but was not very effective. Many of the coordination issues were due to the Ministry being a first-time client of the Bank. Limited understanding of Bank procurement, financial management, and operational procedures caused implementation delays. The Project Coordinator's Office needed to be highly proactive and continuously follow up on issues with the other implementing agencies and with the Bank, which was not always the case prior to restructuring.

There were no financial management issues, and early procurement difficulties (particularly handling large numbers of small-value procurements) were resolved as a result of the competency gained.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The results framework of the original project design was weak (ICR page 10). There was an unclear results chain between the development objectives and outcome/output indicators. Some indicators were not well defined and clearly overstated the project's contribution (satisfactory implementation of the NHPP, satisfactory completion of forward looking action plans). These were not the best proxy indicators to show a strengthened Ministry in policy making and management functions.


The intermediate indicators were clearer, but several depended on the activities of other donor partners (reflective of the project architecture). Otherwise, the results framework was clear on the sources, responsibilities, and frequency for data collection, and there were baselines and targets specified (see Annex 2 PAD and Annex 2 Project Data Sheet).

The results framework was tightened after project restructuring, as non-Bank supported indicators were dropped, and Bank-supported activities were consolidated. Attribution was clearer, allowing the project to be more accountable for its activities. A few key outcome indicators were added (submission of policies in human resources, health financing, and decentralization of health services to the Liberian Cabinet; health facilities that have improved in their accreditation scores).

b. M&E Implementation:
One of the goals of the project was to improve evidence-based decision making in the sector. As such, the project provided for strengthening of the monitoring and evaluation capacity of MOHSW and the decentralized health system. All 15 county offices in the country were fully computerized, and information technology staff were trained to use the HMIS systems installed with USAID support. The regularity and reliability of data and its collection improved.

a. M&E Utilization:
The M&E system has introduced an evidence-based decision making process, including at the decentralized health facility level. As noted in the Borrower Comments (ICR, page 39), an important achievement has been "informed decision making by health managers due to regular submission by county health offices of progressively improved quarterly medical, epidemiology and other reports given the computerization of the HMIS." The project M&E system helped track all the indicators for the project and generated the data for the ICR.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
Safeguards policy was satisfactorily implemented. This was a Category B project, and triggered BP/OP 4.01 (Environmental Assessment) which led to the development of an Environmental and Social Management Framework (ESMF) and a Medical Supplies and Medical Waste Management Plan (MSMWMP). There was no new construction nor land acquisition. The Environment and Social Management issues were handled appropriately in the project. However, the Medical Waste Management Plan was only partially implemented at project completion. Medical waste management shelters at the 5 hospitals were completed as planned; and supplies to implement the waste management plan had begun to be distributed to the sites.

b. Fiduciary Compliance:
There were no fiduciary issues, and "financial management" was rated satisfactory in all ISRs. Three audits were conducted with satisfactory results, but no details about timeliness of audits was provided by the ICR.

Early in the project's lifetime, procurement was the biggest obstacle to project implementation. Only 14.11% of the grant was disbursed prior to the midterm review (ICR, page 11). The Ministry’s lack of familiarity with the Bank’s procedures and the challenge of procuring multiple goods and services that did not amount to much in dollar value plagued the project in its early years. By project close, eight procurement staff in the Ministry were trained in accordance with Liberia’s Public Procurement and Concessions Act. As a result of the competency gained, procurement was no longer an issue at project closing, even in a project with multiple small contracts for goods and services. By project closing, procurement was handled directly by the Procurement Unit within the Ministry of Health, with some oversight by the Ministry of Finance.

c. Unintended Impacts (positive or negative):
None were indicated in the ICR.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Significant
The high dependency on donor aid provides a significant risk to generate the medical resources (including human resources, infrastructure and medical supplies), to sustain basic service delivery. 
Bank Performance:
Moderately Satisfactory
Moderately Unsatisfactory
The Bank preparation phase did not sufficiently address the constraints arising from the Ministry being a first-time client of the Bank. Consequently, limited understanding of Bank procurement, financial management, and operational procedures caused implementation delays up the project's mid-term review. The original design and results framework was ambitious and included indicators and targets that could not be attributed to the project.
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The following lessons from the ICR (page 25) are broadly applicable to other emergency projects.


1) Capacity-building in critical areas of project management should be front-loaded when the capacity of implementers is low. At the start of the project, front-loaded capacity building activities (in areas such as procurement, disbursement, financial management, and M&E) should be planned and budgeted, and implemented strategically, along with carefully planned follow-up activities.

2) The results framework should include indicators that are directly linked to the project, within the purview of the implementing agency, and measurable. In this case, there were indicators affected by a range of factors outside the scope of the project. As a result, it is difficult to draw a results chain directly from the project’s activities to its outcomes and impact measured by the project’s formal indicators.

3) Innovations in design, particularly as they related to capacity constraints, should be encouraged in fragile, post-conflict countries. Successful project implementation in countries with high security problems and no experience with Bank operations is dependent on having strong technical assistance provided by experienced specialists. Innovative approaches to addressing this problem are needed.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is very candid in its assessment of the project's achievements and deficiencies. The inclusion of the Borrower Comments gives an enhanced understanding of project challenges, the low baseline, and ultimately the project's accomplishments.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions