Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Sv - Earthquake Emergency Recovery & Health Services Extension Project


  
1. Project Data:   
ICR Review Date Posted:
12/24/2012   
Country:
El Salvador
PROJ ID:
P067986
Appraisal
Actual
Project Name:
Sv - Earthquake Emergency Recovery & Health Services Extension Project
Project Costs(US $M)
 165.7  175.87
L/C Number:
L7084
Loan/Credit (US $M)
 142.6  142.6
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
 0  0
Cofinanciers:
n/a
Board Approval Date
  12/04/2001
 
 
Closing Date
04/30/2007 06/30/2011
Sector(s):
Health (95%), Central government administration (5%)
Theme(s):
Health system performance (25% - P) Natural disaster management (25% - P) Decentralization (24% - P) Population and reproductive health (13% - S) Nutrition and food security (13% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Moritz Piatti
Judyth L. Twigg Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
According to the Project Appraisal Document (PAD, p. 2) the development objectives were:

" to restore hospital operations and minimize losses in health status to vulnerable populations living in the country's earthquake damaged central and para Central Regions and to improve the health status of underserved populations elsewhere, with special emphasis on the poverty-stricken Northern region."

According to the Loan Agreement (Schedule 2), the objectives of the project were:
"(a) to reconstruct and improve the borrower's health sector infrastructure damaged or destroyed by the earthquakes; (b) to extend the coverage of the borrower's health and nutrition services; and (c) to strengthen the institutional capacity of Ministry Of Health (MOH) to develop and implement policies and priority programs for the health sector."

This review will use the PAD’s statement, as it points to the outcomes that were intended to be achieved by the output-oriented statement in the lending agreement.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
Component I: "Emergency reconstruction of MOH hospital network in earthquake-affected areas" (appraisal US$ 127.0 million, actual US$ 137.97 million)
This component aimed at: (1) rehabilitating/replacing seven of the largest hospitals damaged by the earthquake; and (2) preventive maintenance through strengthening the MOH's integrated preventive maintenance program and a preventive maintenance program for the hospitals being reconstructed under this component. The three hospitals for rehabilitation were San Juan de Dios in San Miguel, San Pedro in Usulután, and Santa Teresa in Zacatecoluca. The four hospitals to be reconstructed were Maternidad Nacional in San Salvador, Santa Gertrudis in San Vicente, Cojutepeque in Cuscatlán, and San Rafael Hospital in La Libertad. In addition, this component aimed to support the unit within the MOH (Infrastructure Executing Unit) that would oversee the implementation of this component, as well as quarterly technical and financial audits.

Component II: "Strengthening essential health and nutrition services in earthquake-affected and extremely poor areas" (appraisal US$ 16.5 million, actual US$ 19.66 million)
This component aimed to: (a) extend essential health and nutrition services to the Northern Zone, where no MOH providers were present, through the development of public-private partnerships; (b) strengthen MOH's primary care delivery in earthquake-affected areas by financing minor equipment, essential drugs, and medical supplies; and (c) strengthen the MOH's capacity to plan coverage extension, manage contracts and performance agreements, and monitor and evaluate performance.

Component III: "MOH institutional development for policy formation, national priority programs, and support systems" (appraisal US$ 16.0 million, actual US$ 10.52 million)
This component sought to strengthen the capacity of the MOH to perform stewardship functions related to quality enhancement, health promotion, public health programs, disease surveillance, regulation, and performance-based monitoring and evaluation. These activities wee to be contingent on the implementation of an institutional strengthening and decentralization strategy to which investments were to be linked. This component had three sub-components: (1) institutional modernization; (2) investments to strengthen MOH national priority programs; and (3) investments to strengthen MOH support systems.

Component IV: "Project management" (appraisal US$ 4.8 million, actual US$ 6.27 million)
This component aimed to: (a) support project management for Components II and III, including the design and implementation of a system to monitor and supervise these two components (monitoring and supervision of Component I was to be financed through Component I); (b) support the impact evaluation of the project as a whole, including Component I; (c) conduct annual financial audits of the project; and (d) finance consultant services, facilities, and equipment for the maintenance of the Project Coordinating Unit (PCU).

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost
Actual total project cost was 6.1% higher than anticipated. This was chiefly driven by a large overrun for Component I, where the costs of rehabilitation and reconstruction were underestimated. The delay between original cost estimates and start of construction meant: (1) an increase in the price of construction materials; (2) an introduction of new seismic norms; and (3) an introduction of new operating and functional criteria. Additionally, modifications to hospital designs and lawsuits had adverse costing implications (ICR, p. 26). Furthermore, substantially less was spent on MOH institutional development, and a strategy for human resources (HR) and information technology (IT) was only partially developed and implemented.

Financing
The project was financed by an International Bank for Reconstruction and Development (IBRD) loan in the amount of US$ 142.6 million (PAD, p. 12).

Borrower Contribution
Against a planned US$ 23.1 million (14% of total project cost), the Borrower actually contributed US$ 33.5 million (19% of total project cost). This was because extra resources were needed to complete construction of Usulután, Zacatecoluca and San Vicente Hospitals. The Government allocated US$ 11.55 million more to cover legal fees due to arbitrations and settlements (ICR, pp.11, 42).

Dates
The Loan Agreement was amended four times to extend the original closing date from 04/30/2007 to 06/30/2011.
The first amendment, in September 2003, reflected the results of negotiations between the Government and the National Assembly. The second amendment, in March 2006, reflected a reallocation of proceeds. The third amendment, in April 2007, extended the project closing date by 18 months to 10/31/2008 so as to give the project enough time to implement the agreed activities. The fourth amendment, in February 2008, involved a third reallocation of proceeds. The fifth amendment, in October 2008, extended the closing date to 10/31/2009. The sixth amendment, in October 2009, both extended the closing date to 03/30/2010 and granted a reallocation of proceeds. The seventh amendment extended the closing date to 10/31/2010, and an eighth one, in October 2010, extended it to 06/30/2011. The ninth and last amendment granted a reallocation of proceeds in May 2011. (ICR, p.56)


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Substantial. . The development objectives of "restoring hospital operations and minimizing losses in health status to vulnerable populations living in the country's earthquake damaged central and para Central Regions and to improve the health status of underserved populations elsewhere, with special emphasis on the poverty-stricken Northern region" were and remain substantially relevant to country conditions at the time of appraisal, and to Government and Bank strategy. At the time of appraisal one third of the population was estimated to have limited or no access to medical care, with the majority of need concentrated in the country’s Northern Region. Many health indicators were below the Latin America and Caribbean (LAC) region’s average, with particular concern for communicable and infectious disease, malnutrition, diarrhea, and respiratory infections. The Bank's current Country Partnership Strategy (CPS, 2009, current at project closure) contains a focus on improved nutrition and access to basic health care services (pp. 11, 21). A World Bank health sector reform project with Components II-IV was already in the pipeline at the time of the earthquake, and was augmented with Component I as an emergency measure to rehabilitate and reconstruct health infrastructure damaged or destroyed by the earthquakes. The two earthquakes struck in early 2001, killing 1,260 inhabitants. The greatest damage was recorded in the departments of San Vicente, La Paz, Usulután, San Salvador, and Cuscatlán and was estimated to amount to 12 percent of the country’s GDP in 2000. Two MOH hospitals suffered severe damage, and another six hospitals bore sufficient impact to require full or partial evacuation. In total, 113 of 361 facilities were affected, representing 55 percent of MOH’s supply of health care services. The earthquake rendered 2,000 hospital beds out of service, reducing inpatient capacity by 25 percent (PAD, p.4)

b. Relevance of Design:
Substantial. The PAD's results chain is plausible. The reconstruction and refurbishment of hospitals in earthquake affected areas credibly feeds into the objective of restoring hospital operations. Similarly, improved health status is conceivably achieved through extending coverage of essential health and nutrition services in the impoverished northern region (through strengthened primary care delivery and provision of essential drugs) and, in the long term, improved institutional capacity.


4. Achievement of Objectives (Efficacy) :

To restore hospital operations and minimize losses in health status to vulnerable populations living in the country's earthquake damaged central and para Central Regions: Modest

Outputs
Six of the seven targeted hospitals were fully reconstructed or refurbished and are fully functioning as planned. Three hospitals (Santa Gertrudis in San Vicente, Cojutepeque in Cuscatlán, and San Rafael in La Libertad) were reconstructed and are fully equipped. Three hospitals (San Juan de Dios in San Miguel, San Pedro in Usulután, and Santa Teresa in Zacatecoluca) were rehabilitated. The planned maternity hospital in Sal Salvador was not built, but the loan financed preparatory studies and some equipment.

All hospital designs were completed, and all hospitals (including the maternity hospital) purchased equipment. A maintenance program was launched in six out of seven reconstructed/refurbished hospitals, meaning that: (a) equipment operators were trained and operating budgets increased; (b) maintenance manuals and trainings were prepared; (c) electrical and heating equipment were upgraded; and (d) generators were installed for medical equipment.

The ICR provides no evidence of the restoration of the 2,000 hospital beds lost during the earthquake or return of hospital inpatient capacity to pre-earthquake levels. The Task Team subsequently added that the project restored 1,366 beds of the 2,000 hospital beds lost during the earthquake, which are fully functional in a more efficient way than before the earthquake as hospital systems were significantly improved. There was thus no need for the additional 612 beds.

Investments were prepared and executed in waste management and disposal, environmental health, and communication strategies. These were only partially achieved in Information Technology and Human Resource Management Systems.

Two functioning HIV testing and counseling centers were set up, meeting the target. A fully functioning web-based information system was developed at MOH, with data on morbidity/mortality rates, HIV surveillance, family records, vectors, and medical supplies.

Outcomes

Data on length of stay were not available for 4 out of the 6 hospitals, since civil works were finished shortly before project closure. The Task Team subsequently added that average length of stay improved slightly to 3.9 days in Cojutepeque and San Rafael Hospitals, which is within the international benchmarks, and that hospital users reported an unspecified reduction of waiting time and improved access to an array of diagnostic tests.

The Task Team also subsequently added that national estimates for infant and neonatal mortality decreased from 15.9 to 7.12 and 11.1 to 4.47 per 1,000 live births respectively between 2001 and 2009. However, given that at the time the majority of civil works were still ongoing, attribution for these outcomes remains ambiguous.

The Task Team also subsequently added that the ratio of inpatient care to ambulatory consultations decreased from about 36 in 2003 to 28 in 2009, showing an increased usage of cheaper ambulatory services.


To improve the health status of underserved populations elsewhere, with special emphasis on the poverty-stricken Northern region: Modest

Outputs:
An institutional strengthening and decentralization strategy was adopted and implemented in all 5 regions, meeting the target.

A Financial Management Unit was established within the MOH Planning Unit for contracting and managing performance agreements, meeting the target.

The project financed the establishment of: (1) a Health Information Unit in the MOH, which monitored basic health services; and (2) a unit for M&E of 12 priority programs, exceeding the target of establishing a Monitoring and Evaluation (M&E) Unit with the capacity to monitor basic health services and at least two priority programs.

The Human Resource (HR) Information System was strengthened through: (a) workshops and study tours on preventive/curative practices; (b) computers and software for MOH central and local levels and hospitals to improve personnel databases and recruitment; (c) support for regulations on MOH internal control of personnel; (d) a manual to train facilitators; and (e) non-economic incentives regarding mental/occupational health workers. This fell short of the target of a strategy for human resource management being designed, approved and implemented.

A new health code and proposal to organize the national health system were prepared and a national health law and regulations were approved, exceeding the target of a reformulated health code prepared.

The MOH launched and implemented strategies to improve care in obstetrics, gynecology and pediatrics in 30 hospitals, exceeding the target of 5.

The project promoted new waste disposal regulations, and helped get a national policy approved and implemented across the public health system (in all public hospitals and 102 primary health clinics), exceeding the target of 7 hospitals and 7 primary health clinics.

Government financing for basic health services in the Northern Region increased. By 2010 the MOH had absorbed the cost of 2,266 more staff and of service delivery into its regular budget. For this purpose, the Ministry of Finance allocated nearly US$ 19 million of additional resources.

284,000 people in the Northern Region, and 352,000 in earthquake-affected areas, were reached with basic health and nutrition services, exceeding the target of 150,000 and 200,000, respectively.

In 2008, NGO teams facilitated an average of 7 visits a month per community and 2.6 per capita contacts a year. . MOH teams made an average of 2 visits per community and 1.6 per capita visits in 2008, the latter of which rose to 3.5 by 2010. Since no baseline data were made available (original or Red Solidaria), no information on progress can be inferred. The target was a 50% increase.

6,000 primary care providers (MOH and NGO) were trained and equipped in delivery of basic health and nutrition services.,NGO providers received limited training. The target of fully trained and equipped government and non-government providers was partially met.

8 contracts were signed with 5 NGOs to provide basic services to cover at least 150,000 people, meeting the target of contracts signed to cover 150,000 people.

17 performance agreements reaching 352,000 beneficiaries were signed to implement outreach programs for the provision of basic health and nutrition services, exceeding the target of 6 performance agreements signed.

A Dengue surveillance system was established and the program implemented, meeting the target. A communication strategy on the prevention and control of HIV/AIDS, STIs and Dengue were developed and implemented via radio and TV campaigns, meeting the target.

Since 2005 all pregnant women were tested for HIV and all HIV-infected pregnant women received free treatment, exceeding the target of 25% identified as infected. All women seeking STI treatment were offered HIV tests, which exceeds the target of 50%. This was done with the assistance of the Global Fund To Fight Aids, Tuberculosis and Malaria (GFATM).

Outcomes:

A substandard baseline survey was discarded, meaning that no baseline data are available for health sector indicators. Progress can, however, be measured using indicators from the Red Solidaria Program in 2007 (Red Solidaria, launched in March 2005 by the government, was a program to provide assistance to extremely poor families, implemented in the same poor municipalities that were part of the project). However, these data come with the caveat of a two-year time lag, and indicators are not the same as the ones originally envisaged.

Community awareness of Aedes Aegypti mosquito breeding control mechanisms increased from 72.7% in 2007 to 93.3% in 2010, exceeding the target of a 25% increase (91%).

DTP (diphtheria, tetanus, and pertussis) vaccination coverage rates have increased from 73.6% to 86.3% since 2007. The original target was a 20% increase from 2005 in the target population.

Pregnant women who receive five prenatal visits, iron supplements and tetanus vaccinations increased from 76.6% in 2007 to 79.2% in 2010. The original target was a 20% increase of at least two prenatal checkups from 2005.

The number of children attending five growth promotion sessions by age 1 increased from 86.3% in 2007 to 93% in 2010, exceeding the target of 90%.


5. Efficiency:

Given a 10-year horizon, the PAD estimated the Internal Rate of Return to be 33.5%, the Net Present Value US$ 123 million, and the Benefit-Cost Ratio 1.76. Benefit estimates were based on: (a) reduction of average length of stay and improved operating efficiency; (b) cost savings through improved general health conditions; and (c) productivity improvements arising from a reduction in mortality and morbidity rates. Sensitivity risk analysis estimated that a 3-year delay, or 20% reduction in benefits, would bring the Internal Rate of Return down to 17%.

The ICR does not include a comparable computation at project completion. It provides no evidence of a reduction in length of stay at hospitals, improved operating efficiency, cost savings through improved general health conditions, or productivity improvements arising from a reduction in mortality or morbidity rates.

Efficiency gains were also expected due to economies of scale by adding the emergency component to an existing project. However, civil works cost 37% more than budgeted for the six hospitals due to an underestimation of prices, new seismic norms requiring different and more expensive construction materials, and expensive ex post modifications of hospital designs. Furthermore, the 4-year, 2-month project delay contributed to increased project management costs.

A costly works supervision firm (ESEO) performed very poorly and allowed the operation of non-performing firms.
Five out of six contracts were awarded to joint ventures with no history of working together, resulting in inefficiencies. The MOH faced four lawsuits from construction firms, costing US$ 9.73 million in arbitrations and settlements. One is still pending, which is expected to amount to a further US$ 2.4 million. The MOH managed to recover US$ 2 million in legal action against poorly performing firms.

Efficiency is rated modest.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
33.5%
100%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The importance of the project given the country circumstances and sector priorities, in combination with a plausible results chain, renders relevance of objectives and design as substantial. Six out of seven hospitals were refurbished/reconstructed with an active maintenance program in place, and these are fully functional. However, little attributable evidence is presented that these contributed significantly to the reduction of losses in health status to the vulnerable populations living in the country's earthquake damaged regions. The attribution of national estimates of reductions in neonatal and infant mortality by 2009 remains ambiguous as the majority of civil works was still ongoing at the time. With regard to improving health status of underserved populations elsewhere, most output targets related to improvements in health status were met, but there was ambiguity in attribution due to a lack of data matching the years of the project (baseline). Furthermore, limited outcome data were provided on health status. As a result, achievement of both objectives was rated modest. Efficiency was modest due to problems with the implementation of Component I.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

The government has shown commitment to sustainability through an average budgetary increase of 75% in all 6 reconstructed/refurbished hospitals, coupled with a thorough maintenance program with training for equipment. Personnel were trained in different areas such as purchasing health services, and medical waste management.

The main achievements in terms of coverage of health service delivery were consolidated by the MOH through the absorption of the cost of 2,266 additional staff and service delivery into its regular budget, towards which the Ministry of Finance has allocated nearly US$ 19 million in supplementary resources. Additionally, vector-transmitted disease programs have been strengthened via community awareness and medical personnel training, policy guidelines for HIV/AIDS and dengue were developed, a new national policy on waste management/disposal was approved for which 90% of health personnel were trained, a human resource strategy was developed, a health code and proposal to organize the national health system were prepared, and a national health law and regulations were approved, along with a comprehensive primary health care model.

Given that the government has shown commitment to progress towards the achievement of the objectives in terms of maintenance, staff allocations, training, additional funding, and policy adoption, Risk to Development Outcome is rated Negligible to Low.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:
The Bank responded rapidly to the government’s emergency request for support to reconstruct the health sector damage caused by the earthquake. An Emergency Recovery Loan Instrument was used to design Component I. However, the Bank and Borrower jointly decided to have a single operation as a Specific Investment Loan and to add the infrastructure component to a health project that was already under preparation when the earthquake hit. The time and resources provided during project preparation did not allow for a thorough, realistic assessment of the hospitals’ damage and health sector needs, which resulted in an underestimation of civil works costs. Additionally, the preparation team was not adequately equipped with the technical expertise the operation demanded. Furthermore, learning from other projects was limited, as there was little prior experience with the merger of two such components.

The overall 'modest' risk rating at time of appraisal was optimistic. Though extensive preparatory work for Components II and III was conducted, the project benefited from high political and sectoral stakeholder involvement, and lessons from a previous earthquake reconstruction project in El Salvador were taken into account, risks of delayed congressional approval, awarding a contract to the management firm, arrangements to implement project components, and insufficient leadership, oversight capacity and technical support provided by MOH all materialized. There was a lack of stakeholder analysis or political assessments that could have led the Bank to anticipate insufficient Government support for modernization of the sector and progress in human resource management systems. Furthermore, the choice of joint ventures to implement construction and the complexity of combining two projects in one were not seen as risks and ultimately contributed to an underestimation of the time and funds needed to complete the project.

For the implementation of Component I, a Construction Management Firm (ESEO) was chosen under the supervision of the Project Coordination Unit. This was done against the initial plan of having a separate Infrastructure Executing Unit, which would have given the civil works component to a Construction Management Firm, due to pressure from the National Assembly. However, this was done without an adequate capacity assessment of the Project Coordination Unit, which proved to be overstretched and was ill-equipped for adequate project supervision.

The Monitoring and Evaluation (M&E) system was overly ambitious. Although M&E occurred, these activities were not implemented as designed (originally the Infrastructure Executing Unit (IEU) and Project Coordination Unit (PCU) were jointly responsible for M&E). Ultimately the PCU monitored all the components, as the IEU was not created. At project launch in 2005, the Project Coordination Unit carried out a baseline study that eventually had to be discarded as it was considered technically substandard (problems with sampling and methods raised questions about the results’ validity). This raises attribution problems at project closure. Also, no health status indicators, or other information on mortality or morbidity, were included in the results framework, meaning that progress towards improved health status or minimized losses in health status, as stipulated in the project's development objectives, cannot be inferred.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
The procurement assessment made during preparation found that the MOH had limited capacity. The original design assigned all procurement activities under Component I to a construction management firm, and thus did not anticipate having to further strengthen the PCU. However, as an Infrastructure Execution Unit was never established, the PCU was charged with M&E for all components. The monitoring capacity of the PCU was weak throughout implementation, and though some M&E occurred, these activities were not implemented as designed (for a more detailed discussion of M&E supervision, see Section 10). Since close supervision was required, the Bank team made frequent supervision missions that were well staffed to assist with resolving bottlenecks. It also appointed staff to supervise the hospital civil works with frequent technical visits and provided support in procurement matters. The Bank team found that there was a difference between the progress reported by the construction firms and the results certified by the works supervision firm (ESEO). It raised the issue on supervision missions, and in 2009 it advised the government not to renew ESEO’s contract based on its poor performance. Shortcomings and delays in executing the civil works as noted during field visits were reflected in Aides Memoire and recommended actions were brought to the government.


The key performance indicators were not revised to better reflect achievement of the development objectives.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
Overall, the government showed ownership and considerable commitment to the project in terms of maintenance, staff allocations, training, additional funding, and policy adoption.

However, there were considerable delays in National Assembly approval of the project, and in counterpart funding. The government did not hire a qualified team for the PCU. It applied its national arbitrage law against the Bank’s recommendation, resulting in costly disputes over five of the six civil works and considerable delays in implementation. The government also hired the same works supervision firm for both phases of construction, even though this was against procedure, the firm was performing poorly during the first phase, and the Bank had recommended otherwise.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:
There were two implementing agencies: the MOH and the PCU. The MOH worked closely with the Bank during preparation and after project approval.

Contrary to Bank recommendation, the project design was changed from use of a separate Infrastructure Execution Unit for Component I to a Works Supervision Firm (ESEO) under management (including M&E) of the PCU, due to pressure from the National Assembly before approval. This was done without an adequate capacity assessment of the PCU, which proved to be overstretched and ill-equipped for adequate project supervision and had difficulties managing the complex procurement process, resulting in substantial delays in preparing bidding documents, frequent modifications thereof, and excessive modifications to the contracts awarded. A potential conflict of interest in the relationship between the PCU and ESEO further compromised the PCU’s supervisory capacity. Under pressure from the government to speed up the process, the PCU made payments for progress that was overreported. Furthermore, ESEO took decisions that were outside its mandate and colluded with some construction firms. Following a Bank recommendation, a technical audit was conducted that found inconsistencies between payments made and reported physical progress, resulting in non-renewal of the ESEO contract.

There was insufficient political support at the MOH for progress in human resource management systems or modernization of the health sector. MOH also did not prepare a baseline in the time needed to monitor and evaluate the Project's progress and achievements. It did not work with the Bank to address multiple needed project changes with required amendments to the Loan Agreement and adjustments to the M&E framework.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The original M&E design was intended to separate the restructuring/rehabilitation component from improved access and system strengthening. The Infrastructure Executing Unit (IEU) was to be in charge of M&E Component I and the PCU in charge of all other Components. During the first year of implementation, an action plan to monitor progress and outputs was to be prepared.

Some indicators chosen to measure the impact of the infrastructure component were not adequate. Average length of stay was already considered low, and patient satisfaction could not be measured as four of the hospitals were just finished at time of project closing. The project used data from the 2007 Red Solidaria report to measure progress on improved access to health and nutrition, since the original baseline study was discarded as it was considered technically substandard (problems with sampling and methods raised questions about the results’ validity). Further, the results framework does not include indicators to measure progress on actual health status, one of the project's two development objectives.

b. M&E Implementation:
The majority of M&E activities were not implemented as designed. The PCU was ill equipped and over stretched for this task. For monitoring of the civil works component, the PCU contracted a Works Supervision Firm (ESEO), which grossly overestimated actual progress and gave an inaccurate perception of project performance. This was overlooked by the PCU, which was under pressure from the government to speed up the construction process. The PCU prepared quarterly Financial Management Reports that described the project’s physical and financial progress. In 2009, after a technical audit, ESEO's contract was not renewed.

a. M&E Utilization:
There was no comprehensive utilization of the M&E framework during project implementation. However, the project financed a Health Information Unit within the MOH and helped establish the M&E Unit for a dozen priority programs. The Project Monitoring System was used to gather data, analyze policy options, improve project management, and support investments.

The final evaluation of Component II was helpful in consolidating support for the new MOH Comprehensive Primary Health Care strategy, which incorporated elements from that component.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
The project was rated Category B. The Bank's environmental policy (OP 4.01) was triggered through the medical waste and hospital reconstruction elements, an environmental assessment of the reconstruction process was carried out, and an environmental action plan was agreed upon. Medical waste management activities included reviewing conditions in each hospital and developing guidelines and procedures. A Project Development Objective indicator of having a solid waste management and disposal policy approved and implemented was included in the Results Framework and achieved. Additionally, these policies were also introduced to public hospitals and primary health clinics outside of the project’s scope.

The government complied with the Bank’s environmental safeguard policies

b. Fiduciary Compliance:
At project inception, the government had adequate financial arrangements. It signed a contract with external auditors, and the PCU presented reliable financial information to manage and monitor the project.

Procurement was not considered a risk in the civil works. According to the project’s original design, all works would have been assigned to a Works Supervision Firm (ESEO), and therefore it was not anticipated that the PCU would require capacity building in this area. However, in one contract the following procurement problems emerged: (a) the process was not conducted as envisioned in the Annual Procurement Plan; (b) a contract award was done through national instead of international competitive bidding; and (c) the process was not submitted to prior review according to procurement thresholds established in the Loan Agreement. In another, a bid was evaluated under shopping procedures based on an item list that differed from the one used in the request of expressions of interest. Although the government demonstrated that the deviations were not intended, they still differed from those in the Loan Agreement and caused US$ 928,285.51 to be cancelled from the loan account. Furthermore, the PCU had difficulties managing the complex procurement process, resulting in substantial delays in preparing bidding documents, frequent modifications thereof, and excessive modifications to the contracts awarded.

The ICR does not provide information on whether audits were on time and unqualified.

c. Unintended Impacts (positive or negative):
None.

d. Other:
None.



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Unsatisfactory
The relevance of the objectives and of the design is rated substantial. However, little attributable evidence is presented that project contributed significantly to the reduction of losses in health status to the vulnerable populations living in the country's earthquake damaged regions or improved the health status of the underserved populations elsewhere. Additionally, the lack of data matching the time frame of the project raises issues of attribution. Lastly, there were implementation inefficiencies, particularly under Component 1.  
Risk to Development Outcome:
Moderate
Negligible to Low
All information presented in the ICR indicates lowered risk rather than continued risk. Lowered risk is mainly due to Government ownership, increased staff allocations, training and maintenance programs, and the implementation of policies. 
Bank Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The ICR (p. 20) provides several useful lessons, as follows:

Borrower's ownership and leadership during the life of the project promotes sustainability. Understanding that the project would strengthen health sector performance and primary health care service delivery, the MOH continued contracting with NGOs upon closure of Component II to avoid disruption of service delivery and used its evaluation to consolidate support for the new MOH Comprehensive Primary Health Care strategy, which incorporated elements from Component II.

Working with joint ventures for civil works can pose significant risks. This project is an example of ineffective joint venture (between a local and international firm). There were the following problems: a) the two companies did not have a history of working together collaboratively, which caused discrepancies between designs/works and modifications of contracts; and b) once the contract was awarded, the foreign firm did not have a solid presence and network in the country, which meant they could not effectively communicate with the PCU.

Selection of the appropriate lending instrument and adequate resources is critical to effective project design. The design and implementation experience of restoring physical assets after a disaster shows the importance of assessing the appropriateness of an Emergency Recovery Loan in terms of time and resources, and reassessing the needs and design against the latest sector trends.

14. Assessment Recommended?

Yes
Why?
Given that the short time period since the completion of the project may not adequately allow for a reflection of the health gains resulting from the reconstruction and refurbishment of the hospitals a PPAR at a later date is recommended.

15. Comments on Quality of ICR:

The ICR provides a clear assessment of the project's achievements, although it would have benefited from a more detailed analysis of the contributions of the outputs to the objectives and the Bank's support to M&E of the project.
Additional information on efficiency (especially revisiting the IRR with actual data), information on safeguard compliance and audits, further detail on the discrepancy between planned and actual disbursements on MOH institutional development, and data on control districts would also be desirable.

Furthermore, the quality of the ICR suffered from internal inconsistencies. The IBRD disbursement amount in the summary sheet was reported as US$ 142.6 million, whereas Annex 1 reports US$ 142.36 million. This does not correspond with the appraisal amount, but the ICR does not mention cancellations. The World Bank project portal, however, reports US$ 556,022 of undisbursed funds. Additionally, the actual amounts of total project cost in Annex 1 include inconsistencies.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions