Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Lc Disaster Management Project Ii


  
1. Project Data:   
ICR Review Date Posted:
06/28/2013   
Country:
St. Lucia
PROJ ID:
P086469
Appraisal
Actual
Project Name:
Lc Disaster Management Project Ii
Project Costs(US $M)
 8.9  13
L/C Number:
C3936, C4498, L7238
Loan/Credit (US $M)
 7.5  10.5
Sector Board:
Urban Development
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  06/22/2004
 
 
Closing Date
12/31/2009 01/20/2012
Sector(s):
Flood protection (60%), Central government administration (15%), Other social services (15%), Sub-national government administration (10%)
Theme(s):
Natural disaster management (50% - P) Water resource management (25% - S) Rural services and infrastructure (25% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Dale M. Hill
Robert Mark Lacey Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
This project followed an IBRD/IDA blend project totaling US$6.27 million (Loan 44190, Credit 31510) which was part of the Emergency Recovery and Disaster Management Project, an Adaptable Program Loan for selected countries in the Eastern Caribbean sub-region which are members of the Organization of Eastern Caribbean States.

According to the Project Appraisal Document (PAD, page 4) the project’s development objective was: “further reducing the country’s vulnerability to adverse natural events (hurricane, floods, etc.) through investment in risk management activities. As such, the objectives of the project are to further strengthen (a) infrastructure against the impact of adverse natural events (hurricanes, flooding, etc.) through the implementation of physical mitigation measures; (b) the response capacity in case of adverse natural event (hurricane, flooding, etc) through capacity building, equipment purchase and investment in emergency infrastructure; and (c) the institutional capacity of the various ministries and agencies dealing with disaster management through the provision of adequate facilities, critical equipment, technical assistance and training.”

According to the Loan Agreement (Schedule 2, page 19): “the objective of the Project is to assist the Borrower to continue preparing for, mitigating against, and responding efficiently to, disasters such as, but not limited to, hurricanes, tropical storms, floods, landslides, earthquakes, and storm surges, through physical strengthening of infrastructure, institutional development and training of human resources.”

The Review is based on the statement of objectives in the Loan Agreement as it is more monitorable.

A $US3 million Additional Financing Credit ((IDA-44980) was signed on November 6, 2008. There was no change to the development objectives, although the key outcome targets did change (see Section 2c below).

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 07/01/2008

c. Components:
1. Physical Prevention and Mitigation works (US$ 5.96 million planned; US$ 9.74 million actual)--retrofitting of schools, health centers and bridges, coastal protection works in Dennery, slope stabilization works Including extension of previously piloted community works using "MoSSaiC" participatory design methods) and stockpiling of related needed supplies and training and capacity building for the Technical Service division of the Ministry of Communications, Works, Transport and Public utilities (hereafter MoPW) and the architectural Section of the Ministry of Physical Development, Environment and Housing (MPDEH).

2. Strengthening Emergency Preparedness and Response (US$1.38 million planned; US$ 1.82 million actual) - support of National Emergency Management Organization (NEMO), including construction of Central Warehouse and Emergency Operation Center (EOC), satellite warehouses, the purchase of specialized disaster equipment, installation of water tanks in shelter, and training for NEMO and community stakeholders involved in disaster response.

3. Institutional Strengthening (US$ 0.41 million planned; US$ 0.28million actual) - Technical assistance (TA) to MPDEH for building code training and sensitization; vulnerability assessment and hazard mapping, with initial focus on three flood prone areas; and TA to the Ministry of Finance (MoF) for a study on vulnerability and risk transfer of Government assets (follows inventory of Government assets undertaken under the First Disaster Management Project.

4. Project management (US$ 0.3 million planned; US$ 1.23 million actual) -- for (a) TA and office equipment supplied to the Project Coordination Unit (PCU), and (b) technical audits.

The Additional Financing approved in July 2008 was applied to components 1 and 4. In the case of component 1, there was a net addition to the appraisal estimate of US$2.71 million, which financed: (i) some additional physical works; (ii) recreation areas from beach reclamation activity in Dennery Village, and extension of other anti-erosion works; (iii) disaster mitigation works including drainage, river walls and slope stabilization, in response to hurricane damage; and (iv) technical audits on disaster mitigation works and a scientific analysis of current landslide vulnerabilities. Support to the Ministry of Finance for a study on vulnerability of Government assets for risk transfer was dropped since it would be financed under another project. There was a net addition of US$0.29 million to component 4, which was used to strengthen the capabilities of the PCU to carry out its new obligations (including audit services) under the project.

The following key outcome indicators were added at the time of the effectiveness of the Additional Financing: (a) the percentage of population with access to improved infrastructure; (b) Dennery village protected from storm surge of a 10 year return period (c) National Emergency Management Office (NEMO) operational in new Emergency Operations Center (EOC) and with updated Emergency Response Plan (d) population with access to shelters with acceptable water systems and with access to emergency equipment throughout the territory; and (e) at least five contractors using improved building standards.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost:
According to the ICR (page 31), total actual project costs were US$13.0 million, slightly higher than the appraisal estimate. As noted above, however, there were some variations by component.
Financing:
The project is financed by a combination of a US$3.7 million IBRD loan (IBRD-72380) and a US$3.8 million IDA credit (IDA-39360). Additional Financing in the form of an IDA Credit of US$3.0 million was approved in November, 2008. According to the ICR, the IBRD Loan and IDA Credits were fully disbursed. There were no other external sources of funding.
Borrower Contribution:
The ICR (Annex 1, page 26) reports that the counterpart contribution of US$2.76 million (appraisal estimate US$2.39 million) was made in a timely manner. One condition of effectiveness was that the Borrower pay a portion of the counterpart funds upfront; in fact, the Borrower contributed the whole amount upfront “to bridge the construction cost for the Emergency Operations Center” (ICR, page 22). There was no Borrower contribution expected or provided for the Additional Financing.
Dates:
The original IDA Credit closed on December 31 2009, five months behind the scheduled date. At the time of the approval of the Additional Financing, the closing date for the original IBRD Loan and the new IDA Credit for the Additional Financing was extended by two years to December 31, 2011. The extension was required in order to: (a) incorporate new works to protect more coastal infrastructure and fully develop the public park on reclaimed land; and b) allow time to complete works that had been delayed.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High
The objective is highly relevant to both the Country Partnership Strategy (CPS) for the Eastern Caribbean at the time of project approval (dated June 4, 2001) and the most recent CPS (dated May 3, 2010). The earlier CPS stated on pages 34 and 35) that the two pillars of the previous strategy--stimulating growth and improving competitiveness (pillar 1) and reducing vulnerability (pillar 2) -- would be carried forward, with pillar 2 restated as "building resilience". The most recent CPS (pages 165 and 166) cites the expected significant cost of hurricanes in St. Lucia and factors in their impact on debt sustainability in a series of scenarios, confirming the importance of continued disaster risk reduction measures.

The objective is also highly relevant to country conditions. As the PAD (page 1) notes, natural disasters causing significant damage are frequent enough to pose a challenge to sustainable development in St. Lucia. The country and the region were poorly prepared for these disasters. While the prior Emergency Recovery and Disaster Management Project was deemed to have achieved its objectives, there were still unmet needs. In St. Lucia, the Government needed help to implement measures mandated under its National Hazard Mitigation Policy. which called for St. Lucia to shifts its approach to adverse natural events from solely response and recovery towards disaster risk mitigation and prevention (ICR, page 2).

b. Relevance of Design:
Substantial, with some caveats.
The project's objective was clear and generally measurable. There is a convincing implicit causal chain between the project's activities and outputs and the attainment of the first sub-objective -- emergency preparedness. Enhanced preparedness could reasonably be expected to result from outputs such as a new Emergency Operations Center (EOC) with an updated Emergency Response Plan; new warehouses stocked with provisions and equipment; appropriate training for the National Emergency Management Organization, other involved agencies, and local responders; hazard maps used in concert with slope stabilization projects; technical assistance in territorial planning; and a study on vulnerability and Government asset risk transfer. However, detail is lacking in the project documents on the content of the Emergency Response Plan and on geographic coverage.

The causal chain was also clear between the project's activities and outputs and the intended achievement of the second sub-objective--"mitigation against effects of disasters." Outputs included works to protect Dennery Village, and the conversion of reclaimed beach into a public park contributing to tourism; small mitigation works throughout the territory, including slope stabilization works built under the participatory "MoSSaiC approach"; retrofitting of selected schools and health centers to be hurricane and flood resistant; the formulation and implementation of operation and maintenance plans; and the adoption of a new building code and its application in new works. However, little detail is provided on geographic coverage of the small works; the project team informed IEG that the location of the works not in Dennery were proposed by Government and approved by the Bank, without considering other locations. The team cited capacity limitations to expanded geographical scope.

Many of the activities and outputs discussed above would also support the third sub-objective-- efficient disaster response. However, elements targeting efficiency of response would also include: a) optimal location of the warehouses and shelters, and sufficient participation of local responders in training, to ensure that the vulnerable population can be reached on time; b) early warning systems (these are mentioned on page 1 of the ICR, but with little detail, and the project team did not elaborate); c) locale-specific emergency response plans and regular simulations and refresher training; and d) provision for regular updating of hazard maps to feed into the national and local emergency response plans. These elements are not explicitly covered by the project's activities.


4. Achievement of Objectives (Efficacy) :

(a) To assist the Borrower to continue preparing... for disasters such as, but not limited to, hurricanes, tropical storms, floods, landslides, earthquakes, and storm surges, through physical strengthening of infrastructure, institutional development and training of human resources. Substantial, with caveats
Outputs
  • The Emergency Operations Center, 11 satellite warehouses, retrofitting of schools and health centers (with water tanks), slope stabilization works ("MoSSAIC") and technical assistance and training for the National Emergency Management Office (NEMO), were completed.
  • An Emergency Response Plan was prepared and updated. A non-technical version of the plan was printed.
  • Technical assistance in territorial planning was provided to public physical planners.
  • Flood risk assessments were carried out in Dennery, and landslide risk assessments and mapping were conducted in the 7 communities where MoSSaiC interventions were financed.
  • Drought hazard maps were produced to provide improved understanding of the hazard risks for better planning. (ICR, page 17).
  • The study on the "Vulnerability and Risk Transfer of Government Assets" was not carried out under this project, but under the follow-on Caribbean Catastrophe Risk Insurance Facility operation.
Outcomes
  • During and after Hurricane Tomas, the Emergency Operations Center was actively used as a command center for relief efforts (ICR, page 16).
  • The satellite warehouses and stocks were "instrumental in the relief operations" at the district and local levels (ICR, page 16).
  • Schools served as shelters to host displaced people (ICR, page 16).
  • According to the ICR (page 16), the shelter management and supply management training "demonstrated [its] value" as NEMO staff and member volunteers "successfully" fulfilled their responsibilities during the emergency, such as managing "efficiently and effectively" warehouses, emergency shelters occupied by disaster-affected people, and logistics related to emergency supply and stock. However, no evidence was presented to substantiate this.
  • During Hurricane Tomas, the communication system failed, compromising effective coordination between NEMO and the national emergency committees, and there was no power backup (ICR, p. 16) Also, NEMO did not convene the emergency committee promptly after Hurricane Tomas, as called for in the Emergency Response Plan; a Bank supervision team had to convene the first meeting. Recognizing these shortcomings, the Government is undertaking a comprehensive review of the plan, to respond to lessons learned from the response operation during Hurricane Tomas.
  • The ICR (page 30) reports that vulnerability assessment and hazard mapping was only partially achieved as three out of four planned assessments were carried out. Hazard maps were produced, but not used by any of the implementing agencies; when they were needed in real time, they could not be found.

(b) To assist the Borrower to continue... mitigating against ...[consequences of] disasters such as, but not limited to, hurricanes, tropical storms, floods, landslides, earthquakes, and storm surges, through physical strengthening of infrastructure, institutional development and training of human resources. Substantial.
Outputs
  • The following infrastructure outputs were completed: coastal protection works in Dennery Village, reconstruction of two bridges, retrofitting of 4 schools and 3 health centers to withstand hurricanes and floods, and 79 small mitigation works (4 more than the targeted output). The ICR does not give full detail on the location or beneficiary population, saying only that, of the 79 small mitigation works, the MoSSaiC community-based slope stabilization approach was used in 7 communities, benefitting 261 homes.
  • The ICR (pages 5, 6, 18, and 28) states that 15,000 square feet of additional beach front area was reclaimed as a result of the coastal protection works in Dennery, and the Additional Financing provided for additional works to enlarge recreational facilities and establish an open communal recreational park on the site.
  • A new Building Code was prepared (though not enacted) and building code training and sensitization activities were conducted for 100 inspectors and design engineers.
  • A Maintenance Manual was prepared for the coastal protection structures at Dennery. In addition, as part of the vulnerability assessment of the health centers, a checklist was developed for the Ministry of Health to be used as a guide for routine maintenance, which the Ministry agreed to adopt.
  • The arrangements for stakeholder-led management of the Dennery beachfront park were incomplete at project closure -- the management had not been transferred to the yet-to-be established community-based committee; in the interim, the National Conservation Authority, responsible for all beaches, is in charge of the management of the park (ICR, page 28).
Outcomes
  • According to the ICR (page 28), the percentage of population with access to improved infrastructure increased from 30% prior to the project to an estimated 80%, compared to a target of 70%.
  • The target that 60% of the population would have access to shelters with acceptable water systems and emergency equipment was met.
  • The ICR (pages 11 and 15) cites several sources providing evidence that project-supported infrastructure, even beyond Dennery Village, withstood natural adverse events. First, the coastal protection works in Dennery were effective during two hurricane events in 2007and 2010 (while also designed to withstand storm surges, there was none during the implementation period). Second, a 2010 Bank assessment undertaken after Hurricane Tomas showed that retrofitted schools and health centers suffered no damage, and (unusually) there were no landslides in beneficiary communities. Also, following Hurricane Dean in 2007, an assessment by the United Nations Economic Commission for Latin America and the Caribbean (ECLAC) concluded that little damage was incurred in Dennery Village, largely due to the project's works.
  • The status of the Building Code is unclear: the PAD states on page 6 that a code incorporating disaster provisions was "recently passed," but the ICR, (page 28) states that "building codes have not yet been enacted, but design approvals for new works ...are based on this code.") The ICR reports the outcome indicator related to contractors' use of the code standards as having been achieved; however, the target appears low in absolute terms (five contractors), and the potential impact on vulnerable populations is unclear.
  • Maintenance of the structures is carried out only partially due to limited public resources, “in view of competing needs in other development areas” (ICR, page 11).

(c) To assist the Borrower to continue... responding efficiently to disasters such as, but not limited to, hurricanes, tropical storms, floods, landslides, earthquakes, and storm surges, through physical strengthening of infrastructure, institutional development and training of human resources. Modest.
  • As noted in Section 3b above on relevance of design, there insufficient emphasis in the results framework and a lack of indicators in the M&E system to allow an assessment to be made of the efficiency of the Government’s post-disaster response. No additional evidence of such efficiency is provided in the ICR.

5. Efficiency:

The PAD (pages 10 and 41) states that an ex ante cost-benefit analysis was conducted to determine the economic viability of the Dennery component (30% of the project’s anticipated costs), excluding smaller physical civil works of less than US$500,000. The ex post cost-benefit analysis covered a larger set of activities, including at least some portion of the institutional development components (coverage is not clear), as well as 20% of the costs & benefits of a parallel Economic Union-sponsored project, which the project team stated to be partially responsible for some of the benefits included in the ex-post analysis. The methodology described (with and without scenarios, including averted damages as benefits, shadow pricing, standard base year for costs, etc.) appears to be standard, but some questions remain concerning: a) the exact coverage of the economic analysis (PAD and ICR); b) whether all components using the “averted damage methodology” factored into benefits a 1/10 ratio to represent the probability of a storm surge; and c) a reference to a "revealed preference method" in the ex post analysis (ICR, page 38), which is not explained..
The ICR reports (page 17) net benefits of 8.6 million Special Drawing Rights for Dennery works, double the appraisal estimate; this comparison may not be valid, however, due to differences in coverage. The corresponding economic rate of return obtained is 20 percent, compared to the 23 percent projected at appraisal; both are well above the assumed discount rate of 12 percent in St. Lucia. The separate evaluation of the small mitigation intervention (MoSSaiC), based on a "typical community" in Castries of 20 houses, showed actual (presumably net) benefits of 257,000 Special Drawing Rights (ICR, page 38). The ICR did not analyze other traditional measures of efficiency, such as comparison to benchmark unit costs.

There were some sources of inefficiency. First, according to the project team, there were cost overruns on some contracts and some inefficiency resulting from the Government's not recruiting a supervising engineering for some of the works, as called for in project design (see Section 9 below and ICR, page 22). Second, a considerable and, to some extent unforeseen, effort had to be devoted to resettlement and land acquisition issues, including encroachment (which the Bank resettlement expert determined to have fallen outside the purview of the relevant safeguards policy (OP 4.12). This increased costs for the Project Coordinating Unit, and caused some delays in execution of the Dennery coastal protection and communal park works (ICR, page 12). Third, hazard maps were produced but not used; one study was financed under another project, though it is unclear if this led to inefficiencies; and communications equipment failed to operate when needed.

On balance, efficiency is rated substantial.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
23%
30%
ICR estimate:
Yes
20%
30%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Relevance of objectives was high, and that of design substantial. Efficacy of two of the three objectives is rated substantial and the third was assessed as modest. Efficiency was also rated substantial.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The main risks to sustaining development outcomes are:
  • Technical: Communications equipment, which failed during implementation period needs to be repaired. The issues which led to the failure of power back-up need to be resolved; and all links in the disaster management results chain that determine effectiveness-- early warning systems, preparedness, and efficient response-- need to be tested periodically, and issues identified in the tests need to be addressed. Simulations need to be a part of an updated emergency plan, to regularly identify weak links in the emergency preparedness and response chain, and to take account of new needs. The risk that some, or none, of this will occur is mitigated by the Government having undertaken a review of lessons learned after Hurricane Tomas (ICR, page16).
  • Financial: There are indications that Government is allowing maintenance of some of the civil works to slip. Moreover, the continued risk of adverse natural events, in and of itself, impacts Government budgets (national and local) by increasing the services that need to be provided. Disasters can also impact household and business assets and livelihoods, which in turn affect the economy and thus Government tax revenue. As a mitigating factor, the ongoing Bank-supported Caribbean Catastrophe Risk Insurance Facility should reduce the impact of hazards on Government budgets.
  • Environmental and natural disasters exposure: The ICR points out that, while the civil works performed well in withstanding one type of disaster (hurricanes), they have not yet been tested by other events (for example, storm surges).
  • Institutional factors: The National Emergency Management Office (NEMO) experienced some difficulties in fulfilling its coordination functions, and both the project team and the ICR (page 20) acknowledge that some training for both NEMO and the Project Coordination Unit was not sufficiently specific. Little detail was provided on the coverage or contents of training for local officials and others involved in response efforts, and the ICR's reporting on number of people trained did not include any local officials. Even if improved, training needs to be sustained and refreshed as new information becomes available.

    a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:

  • Project preparation took account of lessons learned. It built on accomplishments from previous operations, following through on the need for further works in Dennery Village, extending capabilities of the National Emergency Management Office, providing for equipment and training at local levels,and strengthening institutions responsible for vulnerability assessment, territorial planning and building codes. The project's activities were in some cases to help the responsible agencies progress beyond planning to implementation and application; in other cases, the geographical scope was extended. Design was consistent with St. Lucia’s legal framework and medium term economic strategy. While alternatives for the component mix in the overall project design were not presented in the PAD, alternative designs for Dennery Village works, were considered, and the participatory process that led to the choice made was described (pages 39 and 47). Also, the project built sustainability considerations into the design—the Bank requested advanced preparation of maintenance plans for all public works improvements under the project, with responsible public entities identified and a budget dedicated for implementation of the plans (ICR, page 8).
  • The preparation team included an appropriate mix of technical staff and sufficient time for preparation and supervision. Upstream readiness of key preparation milestones was assessed before project effectiveness through feasibility studies, preparation of terms of reference, designs, and tender documents. Most importantly, the complete bidding documents, engineering design, and bills of quantities for the largest works in Dennery had been already prepared under the previous project (ICR, page 7).
  • The team relied on implementation arrangements used successfully in previous operations, namely a Project Coordination Unit (PCU) located in the Ministry of Finance. A Project Launch Workshop was organized, a detailed Operations Manual ( its issuance was condition of effectiveness) and further training in fiduciary aspects was provided for the PCU.
  • However, design did not appear to incorporate some key aspects of disaster management good practice. These included (a) operational early warning systems; b) regular simulations at both national and local levels (which might have prevented communications and power failures); c) setting standards on government performance in disaster response with provisions for monitoring performance against the standards; and d) incorporating indicators in the M&E system to signal issues on the adequacy of maintenance provision.
  • Although the ICR states that the components were targeted at the most vulnerable areas, the project team subsequently stated that there was no assessment of needs by geographic area or beneficiary type; instead the Government's targeting was largely accepted.
  • There was no evident provision for cost-recovery for the water and sanitation connections that would apparently benefit private households and businesses in Dennery Village.
  • M&E design was insufficient (see section 10a below).
  • Despite the design features which responded to anticipated implementation issues, institutional capacity issues emerged; it is difficult to judge if more detailed capacity assessments could have better informed design. For example, the Ministry named as responsible for the Emergency Operations Center construction was overstretched and responsibility had to be transferred; some training proved insufficiently explicit; and capacity problems emerged in dealing with safeguard issues related to resettlement.

  • Quality-at-Entry Rating: Moderately Satisfactory

    b. Quality of supervision:

  • There was frequent and close supervision (missions took place twice per year) and a mid-term review was carried out.
  • The Bank team was responsive to requests for strategic guidance and additional training (ICR, page 21).
  • During implementation, and during the discussions on Additional Financing, Bank staff and Government internalized unintended benefits by taking up opportunities to protect more beneficiaries along the coast and to use additional reclaimed areas for an enlarged public park (ICR, page 5).
  • The supervision team's response to Hurricane Tomas was effective (the Bank convened an emergency committee meeting and helped to assess the damage) and made a notable difference to the Government's own response.
  • There was appropriate staffing in response to emerging issues, for example on safeguard issues (ICR, page 21).
  • Supervision reporting was candid and adequately covered progress towards meeting the project objectives.
  • Supervision played a supportive role during the period leading up to project closure by, for example, agreeing to the use of remaining funds to provide for additional retrofitting of shelters (ICR, page 29).
  • While the ICR (page 13) reports shortcomings in financial management and reporting, the project team explained that repeated attempts were made to provide formal training and on-the-job mentoring. Resulting improvements occurred late in project implementation.
  • Shortcomings in M&E design were not fully addressed, and weaknesses persisted even after the mid-term review.

  • Quality of Supervision Rating: Satisfactory

    Overall Bank Performance Rating: Moderately Satisfactory

    9. Assessment of Borrower Performance:

    a. Government Performance:
    The main government agency was the Ministry of Finance (MoF), although other agencies were responsible for the legal and enabling environment for the project.
    According to the ICR (page 22), the Government was actively involved in the identification and preparation of project activities, and worked diligently to enable the Loan and Credit to become effective. The Government provided counterpart funding up front to bridge the construction cost for the Emergency Operations Center (16% of total project cost), and remained highly committed during the project’s lifetime. However, faced with limited funds, Government only partly lived up to its maintenance commitments. (ICR, pages 8 and 11). The ICR also reported that the formal passage of the new Building Code was delayed, although the Code’s standards were reportedly being applied in Government contracts (ICR, page 28).

    Government Performance Rating: Satisfactory

    b. Implementing Agency Performance:
    Project implementation was led by a Project Coordination Unit (PCU) housed in the MoF. The ICR was largely positive about the performance of the PCU:

    • Although the Unit was implementing 3 other projects at the time, "overall, project implementation was always on track and progressed well according to the project execution schedule...always in compliance with Bank policies” (ICR, page 10). Despite delays in some components, all project activities were successfully completed before the closing of the project. In particular, planned outputs for the large infrastructure component were delivered.
    • Beneficiary participation and implementation success were especially strong for the "MoSSaiC" land stabilization works (ICR, page 22).
    • The staffing numbers of the PCU were maintained in accordance with project covenants, but financial management remained problematic until late in the implementation period, despite repeated training from the Bank.
    • The preparation of a Maintenance Manual and regular government plans did not lead to fully satisfactory implementation of maintenance in all cases. (ICR, page 11).
    • The PCU did not use technical assistance provided under component 4 to recruit a technical engineer to supervise civil works, as agreed with the Bank team, but instead used services of the engineer of the Special Projects Unit of the MCWT&PU to review designs, while hiring supervision engineers for civil works (ICR, page 22).The project team reported that these arrangements did not compromise quality, but may have affected efficiency (the project engineer could have provided the needed supervision support for the entire civil works component, rather than hiring new supervision engineers for each civil works).
    • Capacity limitations emerged with regard to implementation of the Resettlement Plan, but with attention from Bank supervision teams, safeguard compliance was ultimately achieved.

    The performance of the government entities responsible for different project activities was mixed:
    • The Ministry of Communications, Works, Transport and Public Utilities (MCWT&PU)): contracting and supervising coastal protection works at Dennery Village and the majority of the small mitigation works under Component 1; implementation of the communal recreational park at Dennery. Performance satisfactory (ICR, page 23)
    • The Ministry of Education and Culture (MoE): contracting and supervising retrofitting of schools. Performed well (ICR, page 23).
    • The Ministry of Health, Wellness, Human Services, Family Affairs, Gender Relations, and National Mobilization (MoH): contracting and supervising retrofitting of health facilities. Performed well (ICR, page 23).
    • The Ministry of Physical Development, Environment and Housing (MOPD): construction of the Emergency Operations Center (EOC) and the implementation of the Resettlement/Land Acquisition Action Plan (RAP); implementation of Component 3 which included studies and assessments related to risk transfer of government assets and the territorial and hazard mapping and building codes (together with the MoF). Performance less than satisfactory (ICR, page 22)
    • The National Emergency Management Office (NEMO): implementing the technical assistance (TA) activities under Component 2. Performance not rated in the ICR.

    Implementing Agency Performance Rating: Moderately Satisfactory

    Overall Borrower Performance Rating: Moderately Satisfactory

    10. M&E Design, Implementation, & Utilization:

    a. M&E Design:
    There were three outcome indicators (PAD, Annex 3, page 19): "(a) the percentage of population protected by key mitigation works on infrastructure and emergency facilities; (b) readiness and capacity of the country's institutions to prepare for, and respond to, emergencies by planning district; and (c) the country's institutional capacities to prepare for, and respond to, disaster emergencies are strengthened and improvements are seen in the individual planning districts." Only the first of these is quantifiable and had a baseline value. M&E design also included what were described as "intermediate results," but which were, in fact, output indicators for monitoring progress of the significant civil works. The Project Coordination Unit (PCU) was designated as "responsible for monitoring and evaluation of outcomes and results."

    b. M&E Implementation:
    According to the ICR (p. 1), “reporting, monitoring and evaluation were done through: (i) Bank supervision missions that were conducted on average twice yearly; (ii) quarterly IUFRs (Interim Unaudited Financial Reports) including reviews of financial reports and procurement plans; (iii) annual audits; and (iv) a mid-term review (MTR) which was conducted in May 2007. The values of the project’s indicators were recorded every six months in the Implementation Status and Results (ISR) reports. These reports demonstrated that the PDO indicators were monitored consistently during the project life." There was also an evaluation of Bank-financed disaster management activities after Hurricane Tomas.
    At the time of approval of the Additional Financing (AF), the results framework was changed, and the three outcome indicators were replaced with five (ICR, page 3): (a) the percentage of population with access to improved infrastructure; (b) Dennery village protected from storm surge of a 10 year return period; (c) National Emergency Management Office (NEMO) operational in new Emergency Operations Center (EOC) and with updated Emergency Response Plan; (d) population with access to shelters with acceptable water systems and with access to emergency equipment throughout the territory; and (e) at least five contractors using improved building standards. These were more measurable than the original indicators.

    Some key factors contributing to achievement of the development objectives were not captured by the formal M&E framework, even as revised -- for example, the quality and effectiveness of emergency plans, communication systems, emergency committees, training activities, operations and maintenance of works, and Government disaster response activities at national and local levels.

    a. M&E Utilization:
    There was little information in the ICR on M&E utilization, except to say that "the PDO indicators were monitored consistently during the project life" (ICR, page 11).

    M&E Quality Rating: Modest

    11. Other Issues:

    a. Safeguards:
    At appraisal, two safeguard policies were triggered: Environmental Assessment (OP 4.01), and Cultural Property (OP 4.11). For part of the works at Dennery, Environmental Category B was assigned, and the PAD reports that an environmental action plan (EAP) was prepared for these works. For small works, a screening process (including community consultation) was established to ensure compliance with the safeguards policies, should environmental issues or cultural property issues arise. The ICR (page 12) cites the mid-term review in saying that implementation adhered to the Bank’s environmental safeguards policies; the ICR does not comment on cultural property issues.

    At approval of the Additional Financing, new works in Dennery triggered the Involuntary Resettlement safeguard (OP 4.12). The works involved a river berm extension and works to raise the elevation of an embankment to protect an additional 50 Dennery households against flooding, requiring private land acquisitions.

    The ICR (page 10) explains that weak safeguards capacity of the implementing agencies (including the Project Coordination Unit) presented challenges: while the EAP was satisfactorily prepared and implemented, the Resettlement Action Plan was delayed in preparation and implementation, requiring specialized supervision support to achieve with OP 4.12, although this was finally attained. Delays in completion of the Dennery recreational park works resulted. An issue regarding encroachment by squatter shopkeepers on reclaimed land at the beachfront was determined by the Bank to be outside the purview of OP 4.12. Ultimately, after almost 2 years delay, the Government paid for construction of new shops 500 meters away from the original location. The project team stated that the delays were due to Government's strict "pro-poor" resettlement policies, over which supervision missions had little influence.

    b. Fiduciary Compliance:
    Financial Management: As part of project preparation, the Bank’s financial management assessment concluded that the Project Coordination Unit (PCU) had in place an adequate budgetary and financial management system, having benefitted from experience in managing Bank projects. According to the ICR (page 13), financial reporting performance was uneven: interim financial reports were in some instances submitted late, and with inconsistencies and errors (for example, payments being incorrectly apportioned). These problems continued throughout much of the project life. The PCU acknowledged the shortcomings, and addressed them by regularly sending its staff to participate in fiduciary training sessions organized by the Bank. The Bank also provided hands-on training to three PCU financial management staff in the preparation of financial reports. While the PCU’s financial management improved somewhat during the last two years of project implementation as a result of these actions, submissions continued to be late, partly due to the PCU’s heavy workload involving multiple Bank operations (ICR, pages13 and 21).

    Annual external financial audits were completed and submitted to the Bank every year, although some were late. The auditors mentioned a number of generic issues in the management letters, requiring the Government’s attention: (i) the project’s accounts were not reconciled monthly in the Government accounts; and (ii) capital revenue to cover expenditure was not recorded in the Accountant General’s Account. The ICR did not report what measures the Government took to address these issues.

    Procurement: A January 2004 assessment of the PCU’s procurement capacity judged the procurement risk to be “average”. Procurement activities would be carried out by the PCU in close coordination with the Government Tender Boards. The PCU at that time was satisfactorily staffed with knowledgeable procurement officers. The Operational Manual would cover procurement procedures, the Standard Bidding Documents would be used for each procurement method, and model contracts for works and goods would be procured on the basis of three quotations or shopping. The ICR (page13) states that, despite some delays, procurement was managed adequately throughout implementation. Some minor capacity issues that emerged were addressed through training. There were no cases reported of misprocurement.

    c. Unintended Impacts (positive or negative):
    The increased development and tourism resulting from the opportunity to use some additional reclaimed land (15,000 sq. meters of sandy beach) for a public park in the coastal area of Dennery (agreed at the time of the Additional Financing) can be considered an unintended impact (ICR, page 88)

    d. Other:



    12. Ratings:

    ICR
    IEG Review
    Reason for Disagreement/Comments
    Outcome:
    Satisfactory
    Satisfactory
     
    Risk to Development Outcome:
    Moderate
    Moderate
     
    Bank Performance:
    Satisfactory
    Moderately Satisfactory
    There were a number of moderate shortcomings in Quality at Entry, including weak M&E design. 
    Borrower Performance:
    Satisfactory
    Moderately Satisfactory
    Government and implementing agency performance were rated respectively as satisfactory and moderately satisfactory in both the ICR and by IEG. Applying the joint IEG/OPCS harmonization criteria leads to an overall rating of moderately satisfactory. 
    Quality of ICR:
     
    Satisfactory
     
    NOTES:
    - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
    - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

    13. Lessons:
    IEG draws the following lessons from the experience of preparing and implementing this project:

    1. It is important to include all necessary components in a comprehensive Disaster Risk Management Plan. Some elements not explicitly included in St. Lucia’s disaster risk management plans may have compromised the effectiveness of the Government’s response during actual natural hazard events. Such elements included: locale-specific risk vulnerability information; early warning systems; regular simulation exercises to test emergency plans and operability of equipment at national and local levels; and provision of adequate operation & maintenance budgets.

    2. A lack of project-specific institutional capacity assessments as part of project design can undermine implementing agency performance.

    3.Training plans not targeted to specific needs based on institutional capacity assessments can have a reduced impact. If related to generic project management requirements (for example, procurement, financial management, M&E), the plans need to be flexible to allow adaptation during project implementation according to needs or gaps identified in the course of implementation. Training’s effectiveness is enhanced by the inclusion of an evaluation or feedback sub-component.

    14. Assessment Recommended?

    No

    15. Comments on Quality of ICR:

    Positive aspects of the ICR were:
    --Concise, robust narrative, which largely complies with the Guidelines in terms of coverage.
    --Thorough presentation of economic analysis and recalculation of the IRR and net present value.
    --Comprehensive discussion of the implementation responsibilities of various agencies, and their respective performance.
    --Discussion of risk to development outcome which focused on sustainability of outcomes, updating and improving on the PAD discussion.

    Shortcomings included:

    --Inconsistencies: most importantly, in the reference to the development objectives in the PAD (third sub-objective);
    --A more critical review of results framework adjustments that came with the Additional Financing would have been useful;
    --Citing as “lessons” what were in some cases project-specific findings and in other cases, late acknowledgement of problem aspects of design that did not follow good practice in disaster risk management;
    --Referring to "MOPD" --one of the implementing agencies-- without defining the acronym;
    --Referring to a "revealed preference method" in the section on economic analysis without definition;
    --Erroneous overall rating for Government performance (See Section 12 above).
    --Discussion of relevance of design in a way that does not follow OPCS Guidelines; relevance of design is judged with reference to the development objectives.
    --Lack of explanation about the reasons for the decision to complete the study of government asset risk transfer under another project (Caribbean Risk Insurance Facility (CRIFF).

    a. Quality of ICR Rating: Satisfactory

    (ICRR-Rev6INV-Jun-2011)
    © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions