Search in:
Go
Advanced Search

Implementation Completion Report (ICR) Review - Health Sector Development


  
1. Project Data:   
ICR Review Date Posted:
04/27/2012   
Country:
Sri Lanka
PROJ ID:
P050740
Appraisal
Actual
Project Name:
Health Sector Development
Project Costs(US $M)
 72.6  94.88
L/C Number:
C4575, CH095
Loan/Credit (US $M)
 60.0  84.2
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
 0  0
Cofinanciers:
Board Approval Date
  06/15/2004
 
 
Closing Date
06/30/2010 12/31/2010
Sector(s):
Health (26%), Other social services (26%), Central government administration (25%), Non-compulsory health finance (18%), Sub-national government administration (5%)
Theme(s):
Health system performance (33% - P) Population and reproductive health (17% - S) Child health (17% - S) Nutrition and food security (17% - S) Injuries and non-communicable diseases (16% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Judith Hahn Gaubatz
George T. K. Pitman Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
According to the Development Grant Agreement (DGA, page 21) and the Project Appraisal Document (PAD, page 5), the project objective was:


    "to improve efficiency, equity and quality of health care by strengthening planning, management and monitoring capacity at the district, provincial and central level with specific focus on supporting preventive care services at the district and divisional level."

The project was restructured in 2009, with the project objective revised as follows:
    "to contribute to improvements in efficiency, utilization, equity of access to, and quality of public sector health services in Sri Lanka, with a particular focus on district and provincial level services."

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 06/21/2009

c. Components:
The original project components were as follows:

1: Support to district health authorities to improve service delivery and outreach (Appraisal: US$40.0 million; Actual: US$35.7 million): This component aimed to support district health programs, particularly with regards to maternal and child health (MCH) and non-communicable diseases (NCDs). Activities included capacity-building of district and provincial staff in planning and management; and refurbishment and equipping of health facilities. Districts were to prepare individual district health plans based on identified needs, with resources allocated according to an agreed formula (encompassing population size, vulnerability, and health status).

2: Support to central programs and hospitals (Appraisal: US$14.0 million; Actual: US$6.5 million): This component aimed to promote synergy across selected central programs and their convergence at the provincial and district levels, as well as address structural deficiencies in the hospital network. Activities (which were to be primarily carried out by the central Ministry of Health (MOH)) included support to the Family Health Bureau for nutrition interventions and improved quality of immunizations; capacity-building for addressing NCDs; modernization of hospital management and development of country-wide service delivery networks.

3: Support to policy making, budget formulation, and monitoring and evaluation (Appraisal: US$8.9 million; Actual: US$1.86 million); This component aimed to strengthen MOH's stewardship capacity and increase the credibility, transparency and accountability of the public health system. Activities included strengthening public expenditure management, organizing Annual Health Forums to review health sector performance and assess priorities; developing a user-driven M&E system; and piloting an environmentally sustainable health care waste management system.

4: Project management (Appraisal: US$6.3 million; Actual: US$4.5 million); This component aimed to support project management including for district health authorities and provincial councils.

The following components were added, the first in 2005 and the second in 2007:

Tsunami Emergency Recovery (Appraisal: US$19.0 million; Actual: US$19.0 million): This component supported the first phase of an emergency recovery and reconstruction program following the tsunami in December 2004. Activities were implemented and monitored through a separate project (Tsunami Emergency Recovery Program - P094205).

Avian Influenza (Appraisal: US$3.5 million; Actual: US$1.7 million): This component supported the establishment of infrastructure for a potential avian influenza outbreak.

After restructuring, the project components were as follows (as the project funds had almost completely disbursed by the time of restructuring, the revised project component reflect only the amounts allocated from the Additional Financing):

1: Support to decentralized health service delivery at the provincial and district levels (Appraisal: US$23.0 million; Actual: US$24.5 million): The aim of this component and the activities remained essentially the same as Component 1 of the original project. However, the allocation included US$12.0 million for the conflict-affected Northern and Eastern provinces.

2: Strengthening the stewardship functions of the central MOH (Appraisal: US$1 million; Actual: US$1.1 million): The aim of this component was narrowed down from Components 2 and 3 of the original project: to strengthening stewardship capacity. Activities were also narrowed down to focus on the development and effective use of evidence and information systems for policy, planning, monitoring and management.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost:

  • The project cost at appraisal was US$72.6 million. Due to the addition of components related to tsunami recovery and avian influenza, Additional Financing of US$24.0 million was approved to allow for implementation of activities under the revised project design. The actual project cost was US$94.88 million.
  • The project restructuring consolidated, and significantly scaled back, Components 2, 3, and 4 of the original design into one component. According to the ICR (page 17), most central level programs faced implementation difficulties due to lack of ownership by the MOH, weak capacity, and coordination challenges.
  • Almost 80% of project proceeds were spent under the provincial level in accord with the project objectives.

Financing:
  • The original Grant was Special Drawing Rights (SDR) 40.2 million equivalent to US$60.0 million.
  • In February 2005, US$19.0 million was reallocated to tsunami relief activities, which were managed under a separate project (Tsunami Emergency Recovery Program).
  • In August 2007, US$3.5 million was reallocated to a new component for avian influenza.
  • In June 2009, additional financing of SDR 16.3 million, equivalent to US$24.0 million, as an IDA Credit was approved as part of the project restructuring to help finance activities associated with the original project design.
  • Over its duration, the project made an exchange rate gain of about US$6.00 million against the SDR.
  • At completion 99.5% of the Grant and 94.6% of the Credit was disbursed. SDR 0.19 million of the grant and SDR 0.88 million of the Credit was cancelled.

Borrower contribution:
  • The Borrower provided US$10.68 million out of the planned US$12.6 million.

Dates:
  • The closing date was extended from June 30, 2010, to December 31, 2010, to ensure completion of activities under the restructured project.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Original: Substantial. The original objective was closely aligned with the priorities of the government’s health sector strategy (Strategic Framework for Health Development 2004-2015) and the 10-Year Development Framework, Vision for a New Sri Lanka. This included strengthening service delivery at decentralized levels, as well as improving capacity for policy-making and resource allocation at the central level. The objective was also substantially relevant to the Bank's Country Assistance Strategy for FY09-12, which identified quality and efficiency of the health sector as key issues. Revised: Substantial. The revised objective remained substantially aligned with government and Bank priorities. However, it was more precisely articulated and focused on specific goals such as increased utilization, while supporting the decentralization of health service delivery.

b. Relevance of Design:
Original: Modest. Design support the implementation of the Health Sector Strategy in several ways: (i)by strengthening the health program delivered by the district authorities; (ii) by improving the capacity of MoH at the central and provincial levels for policy-making and supporting the budget formulation process to reflect priority objectives; and (iii) by strengthening the orientation of the public health system towards outputs and results.Thus the project design encompassed a wide spectrum of activities that were likely to contribute to achieving the stated objectives, including strengthening management capacity at multiple levels of the health sector ministry. However, the range of activities was ambitious and therefore the likelihood of full and effective implementation was highly uncertain.
Revised: Substantial. The range of activities was scaled down to focus on achieving the more specific objectives, including eliminating the more centralized programs for the MOH and hospitals. The renewed emphasis on district activities also supported the decentralization of health service delivery.


4. Achievement of Objectives (Efficacy) :


The original objective was:
    To improve efficiency, equity and quality of health care by strengthening planning, management, and monitoring capacity at the district, provincial and central level with specific focus on supporting preventive care services at the district and divisional level.

The restructured objective was:
    To contribute to improvements in efficiency, utilization, equity of access, and quality of public health service services in Sri Lanka, with a particular focus on district and provincial level.

In the following, achievements are (a) assessed for the original three objectives and geographical coverage; and (b) then assessed for all four objectives of utilization with a particular focus on district and provincial level.

Original Objective:

Outputs
Central level:
  • 22.2% of provincially managed health facilities were practicing a set of techniques to promote workplace organization, ensure adherence to standards, and foster the spirit of continuous improvement, an increase from 2.2% but short of the target of 40%.
  • Nutrition programs were reviewed, with action plans developed. A National Nutrition Surveillance System was piloted in 30 divisions; however, the ICR (page 22) notes that “there is a high risk of failing to sustain nutrition surveillance activities following reallocation of funds. Weak coordination of nutrition activities persists because of fragmentation.”
  • 93 clinical guidelines were developed.
  • A national action plan for NCD prevention was developed.
  • 20 sentinel surveillance sites for Avian Influenza were established, as well as 14 isolation units.
  • Over 500 best practice immunization clinics were established, with capacity to detect adverse effects following immunization. A review of the clinics’ performance was conducted but results were not yet available.

However,
  • The ICR (page 23) reports that “limited progress was made to strengthen the budget framework process at the central level to orient the public health system towards outputs and results and to ensure alignment of public resource allocation to sector priorities.”
  • A number of central level programs faced implementation difficulties due to lack of ownership by the MOH, weak capacity, and coordination challenges.

District and provincial levels:
  • Primary and secondary health facilities in rural areas were refurbished and equipped with medical, diagnostic and IT related items. 92.3% of districts were equipped with a set of life saving equipment for newborns, increasing from 20% and surpassing the target of 60%.
  • The percentage of lower level hospitals with emergency treatment units increased from 13.3% in 2008 to 53.6% in 2010, surpassing the target of 50%.
  • The proportion of MCH clinics with basic facilities increased from 51% to 63%. No target was provided.
  • All districts established accredited Program Management Centers (PMCs) and were submitting annual financial and physical progress reports within six months after the end of each financial year.
  • 96.9% of districts were completing monthly adverse effects following immunization surveillance reports on time, an increase from 80% and surpassing the target of 95%.

Outcomes
Central level:
Improved efficiency - Substantial
  • The percentage of institutional deliveries in teaching/specialist hospitals decreased from 47% in 2004 to 33.9% in 2010, surpassing the target of 40%.
  • The bed occupancy rate at higher level hospitals (General Hospitals and above) decreased from 97.8% in 2008 to 87% in 2010, falling slightly short of the target of 85%, suggesting that there is a lower level of use of specialty hospitals.

Improved equity - Negligible
No specific outputs or outcomes reported for this objective.

Improved quality - Modest
  • The number of cases of adverse effects following immunization increased from 1245 to 5570; the ICR (page 22) reports that this was likely due to improved detection capacity.

District and provincial level:
Improved efficiency - Substantial
  • The bed occupancy rate at lower level hospitals (Base Hospitals and below) increased from 36.7% in 2008 to 46.2% in 2010, surpassing the target of 45%.

Improved equity - Negligible
No specific outputs or outcomes reported for this objective.

Improved quality - Modest
  • Although no specific outcomes are reported, it is likely that the equipping of facilities led to some quality improvements.


Revised Objective:
See above for outputs and outcomes related to the first three objectives of efficiency, equity and quality at the district/provincial level.

District and provincial level:
Improved utilization - Substantial
Outputs
  • 77% of districts conducted at least 10 “well woman” screening clinics, an increase from 47.8% and achieving the target of 75%.
  • Outreach activities to underserved communities and schools were conducted. 1450 estate workers (identified as a vulnerable group) received specialist antenatal care outreach services.
  • In the conflict-affected Northern province, 73 health facilities were reconstructed and antenatal and post-natal services were resumed in 2009 with the provision of equipment and transport facilities to field health workers.

Outcomes
  • The proportion of women over 35 years old being screened for cervical cancer increased from 1101/100,000 in 2008 to 2979/100,000 in 2010, essentially achieving the target of 3000/100,000.
  • The percentage of pregnant women screened for anemia increased from 14.3% in 2008 to 48% in 2010, falling short of the target of 85%.

5. Efficiency:

Modest due to lack of project-specific data. The PAD provides a discussion of the economic rationale for the project i.e. a justification of public intervention in the health sector on equity grounds and on the need to address market failures. However, there is no quantitative analysis of project-specific costs compared to benefits.

The ICR does not provide a detailed economic analysis, but identifies project activities and outcomes that may indicate efficient use of resources, such as focusing on maternal and child health and non-communicable illnesses; introducing a needs-based formula to allocate project funds more effectively; decreasing utilization of higher level hospitals; and increasing utilization of lower level hospitals.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The original project outcome is rated Moderately Unsatisfactory. The relevance of the objectives is rated substantial, while there were shortcomings in design. Achievement of the objective to improve efficiency at the central and district/provincial levels is rated substantial, while achievement of the objective to improve quality at the central and district/provincial levels is rated modest due to limited information on outcomes. The achievement of the objective to improve equity is rated negligible due to lack of information on outputs or outcomes. There was limited evidence of efficient use of project resources.

The revised project outcome is rated Moderately Satisfactory. The relevance of the objectives and design are rated substantial. There was substantial achievement of the objectives to improve efficiency and utilization at the district/provincial levels, although modest achievement of the objective to improve quality and negligible achievement of the objective to improve equity. There was limited evidence of efficient use of project resources.

According to ICR guidelines, the combined outcome rating is determined by weighting the proportion of the loan that disbursed before and after project restructuring. While the IDA grant for the project was reduced by the US$19.0 million allocated to the Tsunami Emergency, these funds were replaced with the additional financing of US$24.0 million. The net IDA financing for the project was thus US$65.0 million.) As the loan had disbursed US$59.0 million out of US$65.0 million, or 90.1%, before restructuring, the weighted outcome rating is Moderately Unsatisfactory.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

A follow up health project is under preparation, which will address the same health challenges in the country and reflects the continued commitment of the government. The long-term military conflict has come to an end, decreasing the risk to improved health outcomes in the conflict-affected areas.

Key project activities (such as establishment of emergency treatment units and planning units at the district level) were effectively institutionalized within the health sector, although the sustainability of other activities (such as nutritional surveillance) is less certain due to lack of funding or prioritization. Weak capacity of the MOH was identified as a high risk during project preparation; while mitigation measures have had some effect, adequate capacity in the MOH and other implementing units remains an issue.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:
The project design was built on a detailed assessment of the health sector, and consultations with major stakeholders were conducted to build support. The district-level component was designed to disburse funds in a programmatic manner (using government fiduciary arrangements to finance annual district health plans), a feature which provided districts with flexible and predictable funding.

However, the project design included numerous sub-components that were to be managed by 15 different departments. Inadequate capacity was identified as a high risk, but mitigation measures were insufficient. The ICR (page 11) notes that the inclusion of "already well performing programs e.g. MCH and immunization, was questionable. Emphasis should have been placed on under-performing programs/areas e.g. NCDs, nutrition, health care waste management, health systems improvement actions to optimize health care performance." The project team clarified that while these programs were performing well with respect to coverage, there were quality and equity issues that still needed improvement. A Project Management Secretariat (PMS) was established in lieu of a traditional project management unit, and while this arrangement was intended to facilitate fiduciary matters, the placement of the PMS outside the regular MOH structure made it difficult to exercise technical oversight or mainstream and coordinate activities within the sectoral ministry. The M&E framework was inadequate, as the selection of outcome indicators were not appropriate given the time frame of the project and attribution issues.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
Supervision missions were appropriately staffed and considerable efforts were made to address fiduciary capacity, leading to overall satisfactory fiduciary performance by the end of the project. Critical changes were made through the project restructuring, including sharpening of objectives and the M&E framework, simplifying the project design, and absorbing the PMS into the MOH to strengthen project oversight. These changes led to significant improvements in implementation performance.

However, the project restructuring was not finalized until 2009, one year before the original closing date. The ICR (page 7) also notes that the reallocation for tsunami and avian influenza activities took place "without a clear plan on what activities to cut, creating uncertainty regarding funding and disrupting implementation of central level activities." The Borrower's ICR (page 39) notes that frequent turnover of Bank task team leadership caused difficulties, including making frequent modifications to planned activities "based on personal opinions and impressions."

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The government sustained its commitment to the project objectives, as well as provided counterpart funding in a timely manner. However, there were delays in appointing staff to critical positions and the ICR (page 7) reports that the delayed restructuring was in part due to "the uncertainty over the Government's willingness to replenish the shortfall of US$22.5 million reallocated to tsunami and avian influenza response." The project team reported that the initial unwillingness was due to the fact that while the original project was funded as a Grant, the Additional Financing was provided as a Credit.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
Provincial and district level health units effectively implemented their activities, including preparation and execution of district health plans, and alignment of district plans to provincial budgets. Fiduciary performance was initially inadequate but improved significantly by the end of the project. However the performance of the MOH, which was responsible for the central program activities, was characterized by "lack of ownership... weak implementation capacity, and coordination challenges" (ICR, page 17).

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The original M&E design included detailed monitoring arrangements, as well as data collection activities and evaluation activities. This included an evaluation of the district health plan program part-way through the project period with the intent to refine the design based on lessons learned. However, the M&E framework was based on indicators (i.e. infant mortality rate and maternal mortality rate) that were inadequate given the time frame of the project and attribution issues.

b. M&E Implementation:
The M&E framework was revised during project restructuring to include indicators that were more directly relevant and attributable to the project activities, such as bed occupancy rates and screening for specific conditions.
The ICR (page 9) reports that while the original intermediate outcome indicators were not tracked, the revised indicators were regularly monitored. Some data collection activities (national nutrition surveillance) were implemented, although given the numerous implementing units and the lack of qualified staff, there was overall inadequate implementation of M&E activities. The ICR (page 9) also notes that an M&E task force created to improve coordination across the different MOH directorates was not able to resolve the problems that persisted throughout the project period.

a. M&E Utilization:
The ICR (page 9) reports that while substantial amounts of data were collected, the actual use of information for planning and management was limited.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
The project was categorized as an Environmental Category "B" project under OP 4.01 Environmental Assessment, and a health care waste management plan was prepared. Project restructuring documents (Project Paper, page 12) report that supervision missions up to that point noted satisfactory implementation of the plan. The project team reported that while government initially did not prioritize environmental matters, compliance was eventually satisfactory. Two centralized steam sterilizers were installed to treat medical waste from the 10 largest teaching hospitals in the Colombo Municipal area and are partly operational. The ICR does not report whether safeguard measures were updated due to the expanded scope of the project.

b. Fiduciary Compliance:
Financial management: Project restructuring documents (Project Paper, page 11) note that a brief FM capacity assessment was conducted for the restructured PMU and found the arrangements to be satisfactory. However, financial management of the district component was unsatisfactory at the beginning of the project but improved to satisfactory in 2009 after the main problem of lack of qualified staff was addressed. The ICR (page 10) reports that the financial management arrangements of the central MOH components were generally in compliance with covenants, with occasional delays in preparing financial monitoring reports. One exception was a qualified opinion on financial statements from the Auditor General. The project team reported that the issues were subsequently resolved.

Procurement: The ICR (page 10) reports that procurement performance at the central level was mostly satisfactory. Provincial level units were initially characterized by weak capacity but capacity building efforts by the Bank team improved performance. One procurement shortcoming noted was the poor packaging of contacts, especially for civil works, as procurement plans were tied to budget cycles and led to piecemeal execution of the civil works.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Unsatisfactory
The original project outcome is rated Moderately Unsatisfactory. There were shortcomings in design, and achievement of the objective to improve quality is rated modest due to limited information on outcomes. The achievement of the objective to improve equity is rated negligible due to lack of information on outputs or outcomes. There was limited evidence of efficient use of project resources. The revised project outcome is rated Moderately Satisfactory. There was modest achievement of the objective to improve quality and negligible achievement of the objective to improve equity. There was limited evidence of efficient use of project resources. As the loan had disbursed US$59.0 million out of US$65.0 million, or 90.1%, before restructuring, the weighted outcome rating is Moderately Unsatisfactory. 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Moderately Satisfactory
Moderately Unsatisfactory
The project design was complex with numerous sub-components; mitigation measures to address inadequate capacity were insufficient; the M&E framework was inadequate, as the selection of outcome indicators was not appropriate given the time frame of the project and attribution issues. 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The following lessons are drawn from the ICR:
  • Decentralized health services work best when roles and responsibilities are clear, the central level fulfills its stewardship functions, and the decentralized levels are provided with adequate resources. The programs with well established structures and positions at provincial and district levels, such as MCH, registered better performance compared with centrally managed ones, without similar structures.
  • As discussed in the ICR (page 17), while reallocation of project funds to non-project-related activities is justified in emergency contexts, there should be a clear idea of how the reallocations will affect planned project activities and how the shortfalls will be addressed.
  • M&E activities among multiple implementing units should be well coordinated to ensure optimal use of M&E data.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

Although sufficient data and analysis were provided in the ICR to enable assessment of outcomes, the ICR did not adhere to guidelines in assigning ratings for outcomes. A six-point scale (Highly Satisfactory, Satisfactory, Moderately Satisfactory, etc.) is to be used for assessing overall outcome. In addition, ICR guidelines require a separate assessment and rating for the original project outcome vs. the revised project outcome, with a weighted combined outcome rating at the end.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)