Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Brasilia Environmentally Sustainable Project


  
1. Project Data:   
ICR Review Date Posted:
12/31/2013   
Country:
Brazil
PROJ ID:
P089440
Appraisal
Actual
Project Name:
Brasilia Environmentally Sustainable Project
Project Costs(US $M)
 115.3  117.2
L/C Number:
L7326
Loan/Credit (US $M)
 57.6  44.7
Sector Board:
Environment
Cofinancing (US $M)
 N/A  N/A
Cofinanciers:
Board Approval Date
  08/27/2005
 
 
Closing Date
03/31/2011 12/31/2011
Sector(s):
General water sanitation and flood protection sector (70%), Sub-national government administration (25%), Central government administration (5%)
Theme(s):
Environmental policies and institutions (29% - P) Pollution management and environmental health (29% - P) Municipal governance and institution building (14% - S) Other urban development (14% - S) Other social development (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Richard C. Worden
Ridley Nelson Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The project’s objective, as stated in the Loan Agreement (7326-BR), was: “to ensure the supply of quality water resources to meet the demands of the Borrower’s metropolitan area by improving environmental planning and management activities. These activities will be further enhanced by the carrying out of poverty reduction strategies in certain urban areas and the rehabilitation of the environment around key river basins” (Schedule 2, p. 17). This statement of the project’s objective will be used to assess the project.

The Project Appraisal Document (p. 4) stated that the Project Development Objective was slightly different substantively, by adding the statement to meet the growing needs of the Federal District and the Brasilia Metropolitan Region “to reduce regional inequalities.” This aspect of the PDO was not included in the Loan Agreement, and will not be used as part of this project assessment.

The project’s description and some indicators were “redefined” on December 17, 2010 (4˝ years into the project) when the project was formally restructured and the third original PDO Outcome Indicator (strengthened technical and institutional capacity) was dropped from the Results Framework. Other changes were made to scale back activities, such as the waste water treatment facilities in Aguas Lindas intended to improve the water quality of the receiving water bodies of the Descoberto River and closing the existing Jocquei Clube dump site and replacing it with a new sanitary landfill. In addition, changes in the allocation of resources among various different project components, extension of the project, and modification of some of the outcome targets in the area of water resources and solid waste management during the final year of project implementation were included as part of this restructuring. However, the project development objective was not changed.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? No

Date of Board Approval: 12/24/2010

c. Components:

Original Components:

1: Policy and Institutional Development (estimated: US$5.0 million; actual: US$7.3 million). This component supported institutional strengthening and technical assistance to enhance the technical and institutional capabilities of the Government to undertake informed decision-making with respect to metropolitan issues, especially in promoting a more sustainable urban and environmental development. It had three sub-components:

1.1. Territory Development and Land Management. This encompassed four main thematic areas related to territory, urban, housing and land use policies and institutional enhancement.

1.2. Environmental and Water Resources Management. This addressed six thematic areas related environmental and water resources management.

1.3. Environmental Sanitation Management. This addressed four thematic issues related to environmental sanitation services.


2: Social Inclusion and Poverty Reduction (estimated: US$31.2 million; actual: US$70.3 million). This component was comprised of activities to reduce poverty and promote social inclusion of localities with critical social conditions, coupled with actions to abate water resource pollution loads into key river systems. It had two sub-components:

2.1. Integrated Development Project for Vila Estrutural. This consisted of two activities: (i) planning, technical designs and community participation; and (ii) civil works and public participation activities.

2.2. Support to the Joquei Clube Solid Waste Landfill waste pickers (scavengers).


3: Water Resources Protection (estimated: US$71.5 million; actual: US$33.2 million). This component included activities to abate water resource pollution loads and improve the quality of life in the communities of Aguas Lindas and Vicente Pires, as well as protect the Brasilia National Park from improper solid waste disposal activities. It consisted of six sub-components:

3.1. Improving sanitation services in Aguas Lindas

3.2. Improve Water and Sanitation Services in Vicente Pires

3.3. Closure of the Joquei Clube Solid Waste Sanitary landfill

3.4. Construction of a solid waste sanitary landfill

3.5. Rehabilitating environmentally degraded areas

3.6. Waste water sludge treatment


4: Project Management, Monitoring and Evaluation (estimated: US$2.9 million; actual: US$5.1 million). This component supported the creation of the necessary technical, administrative and financial management conditions and capabilities to implement the project.

Revised Components:
Component 1 was altered substantially to include the following changes to the structure and activities of its subcomponents:

1.1 New capacity-building activities were included for the Water, Energy and Sanitation Regulatory Agency of the Federal District (ADASA) while other activities related to the improvement of the Borrower’s territory development and land management policies were removed.

1.2 The development of new management tools for the Secretariat of Urban Development and Environment (SEDUMA) was included, and the National Environmental Institute (IBAMA) was replaced by the Borrower’s Environmental Institute (Instituto Brasília Ambiental or IBRAM) as part of the Government’s decentralization of environmental roles and responsibilities.

1.3 SEDUMA and ADASA were included as beneficiaries of technical assistance activities.

1.4 The sanitation company for the areas adjacent to the Federal District was removed from the project and two new activities were included; (i) technical assistance to improve the management and protection of water sources; and, (ii) environmental impacts caused by the construction of sewerage systems in the Federal District were to be identified and remediated.


Component 3 was also significantly changed in the following ways:

3.1 and 3.5 were removed.

3.3 Revised to include the closure and environmental recuperation of only one-third of the Joquei Clube solid waste landfill, rather than its complete closure as initially planned.

3.4 Revised to include the launch of the procurement process to construct the new landfill, instead of the original target of completing the construction and operation of a new sanitary landfill.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost: Total project cost was estimated at US$115.3 million at appraisal. Actual costs were slightly higher at US$117.2 million.

Financing: The World Bank was expected to provide half of the estimated total project financing of US$57.64 million, but it financed only 78 percent of this estimate, or US$44.74 million mainly due to implementation delays. The actual loan disbursements for Component 2 (Social Inclusion and Poverty Reduction) were more than double estimated amounts (225 percent) and 146 percent of estimates for Component 4 (Project Management and M&E (see Section 2.a above). Financing for Component 1 (Policy and Institutional Development) of US$7.36 million was nearly equal to its estimate of US$7.6 million while Component 3 (Water Resources Protection) reached only 46 percent of its estimated requirement.

Borrower Contribution: The Borrower (the Government of the Federal District) was expected to contribute the same amount as the Bank’s financing (US$57.64 million) but provided just 51 percent of that, or US$29.61 million. However, as part of the second project restructuring agreement with the Bank to extend the project for nine months, the Borrower agreed to assume the financial responsibility for financing the remaining activities not completed by the new closing date. This amount covered the remaining US$42.85 million.

Dates:
The project was appraised by the Bank on March 7, 2005 and became effective on May 16, 2006. Early project implementation was slowed considerably during 2006 because of local elections in the Federal District. As a result of these elections, a major institutional reshuffling was carried out by the new District Government, which requested an amendment to the loan agreement to make the project implementation arrangement consistent with its institutional reorganization. The amended loan agreement was signed on May 21, 2007 and a new effectiveness letter was signed on July 27, 2007 (ICR, p. 9). The project was restructured on December 17, 2010 with over 82 percent of project funds disbursed (US$36.42 million out of US$44.88 million) to redefine its scope and targets, but not its objectives. A request by the Borrower for a second restructuring in March 2011 was agreed to by the Bank’s Country Director to extend the closing date by nine months from March 31, 2011 until December 31, 2011 to complete several unfinished activities at their own cost.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Relevance of Objectives Rating: Substantial
The relevance of the project’s objectives was consistent with the conditions existing in the country and with the Bank’s Country Partnership Strategy with Brazil in effect at the time of project implementation and closure. At the time of the BES Project’s closing at the end of 2011, it was relevant with respect to the second (Improving the provision of public services for low income households) and fourth strategic objectives (Improving sustainable natural resource management and climate resilience) of the current Country Partnership Strategy between the Bank and the Government of Brazil (for fiscal years 2012-2015). In particular, integrated urban water resources management, improved access to water supply, sanitation and health services, greater recreational opportunities, and slum upgrading remain important development issues in Brazil today. At the time of the project design and appraisal, the BES Project’s objective was consistent with the Country Assistance Strategy (CAS) of FY2004-2007, namely the development of “A More (Environmentally) Sustainable Brazil.” This CAS was followed by a Country Partnership Strategy for fiscal years 2008-2011 in which “Environmental Sustainability” continued to be one of the thematic pillars, including the goal of “improving the environmental and social quality of infrastructure lending” among its eight elements.

b. Relevance of Design:
Relevance of Design Rating: Modest.
The original design of the BES Project was developed in response to a request from the Borrower to the Bank for financial and technical assistance in formulating and implementing a sustainable development strategy for the greater metropolitan area of the capital of Brazil – Brasília, and its rapidly-growing, but unplanned surrounding communities (‘entorno’) in the Federal District and adjoining State of Goias. A comprehensive regional assessment carried out during project preparation was used to select the most cost effective interventions from an urban environmental perspective, the aim was water resource management within a holistic watershed approach. The BES Project was designed to be part of a much broader regional development program, encompassing an Inter-American Development Bank (IDB) sanitation project and a proposed Global Environment Facility grant-funded project to preserve the sustainable productive landscape and biodiversity of the Cerrado. However, the latter project “did not occur because of a preference to focus on other areas in Brazil” (ICR, footnote 1, p. 2). The design of the Results Framework was sound and logically linked to the objectives and the three thematic components and activities of the project with its objectives. The comprehensive nature of the design for the BES Project had been based in part upon ‘lessons learned’ from similar urban upgrading projects in other Brazilian municipalities.

However, the project’s design was unrealistically optimistic and made it highly unlikely that the project’s objectives would be achieved within the project’s time and resource parameters. The comprehensive, ambitious scope of the BES Project suffered from thematic overreach (by including both solid waste and water resource management activities and goals) and it required unmanageable levels of inter-agency coordination and cross-jurisdictional collaboration to successfully achieve its stated goals.

Partly as a result of the design problems, the project was restructured in December 2010 and several activities and targets were scaled back to simplify implementation and complete work already undertaken. For example, solid waste management activities were simplified by: (i) eliminating public sanitation works in one of the two targeted communities (Aguas Lindas), (ii) dropping water quality condition improvements in one of the two targeted river watersheds (the Descoberto River), and (iii) providing more technical assistance to implementing agencies. This revised design made it more likely that the project could realistically achieve its overall objective. However, 82.5 percent of disbursements had already been made by this time, with only one year remaining to implement these activities.


4. Achievement of Objectives (Efficacy) :

The objective of the BES Project was: “to ensure the supply of quality water resources to meet the demands of the Borrower’s metropolitan area by improving environmental planning and management activities. These activities will be further enhanced by the carrying out of poverty reduction strategies in certain urban areas and the rehabilitation of the environment around key river basins.” For evaluation purposes, it will be assessed in terms of: (i) quality of water resources of the metropolitan area by improving environmental planning and management; (ii) poverty reduction strategies carried out in certain urban areas; and (iii) the rehabilitation of the environment around two key river basins.

a) Ensure the quality of water resources of the metropolitan area by improving environmental planning and management. Rated: Modest.

Substantial progress was made on building the wastewater collection system in Villa Estrutural, by far the largest and most critical settlement in the surrounding metropolitan area of Brasilia in both social and environmental terms. Residential hook-ups to collection trunk lines leading to the wastewater treatment plant increased from only 9.2% of households in 2004 with access to sewerage services compared to 92.2% of households by mid-2012.Thus, these civil engineering works were substantially completed, although a very small percentage (<1%) of residential connections are still pending completion.

The second primary investment to abate water resource pollution loads, namely the construction of the sewage system in Vicente Pires, was not finished when the project was closed, and is still pending completion before it becomes fully operational. Thus, while no improvement in water quality of the Vicente Pires River can be demonstrated through water quality sampling tests, substantial progress has been made toward “ensuring the quality of water resources in the metropolitan area.” This is largely attributable to the investments made by the BES Project, and as a result of continued work and funding by the Federal District Government since the project’s closing. (Water quality improvements to the Descoberto River were no longer possible since this was linked to sewerage works in Aguas Lindas, which were dropped from the project in December 2010.)

Finally, the third major source of pollution run-off into the Vicente Pires River was anticipated from the closing of the Joquei Clube sanitary landfill and its environmental recuperation. This activity was significantly scaled back at restructuring to only include closing and cleaning up one-third of the landfill, rather than its complete closure and clean-up as initially planned. This activity was abandoned after multiple attempts to concession the new landfill under a new Public-Private Partnership model of public construction and private management failed. In addition, the ICR noted (p.21) that there were an increasing number of invasions near the Jocquei Clube landfill that continued to put the quality of water resources at risk. The revised Sub-component 3.4 to build a new sanitary landfill was scaled back to just launching the procurement processes for this work, but despite repeated attempts to tender bids for its construction, this was never brought to fruition during the project’s implementation.

b) Poverty Reduction Strategies carried out in certain urban areas. Rating: Modest.

The focus of the majority of poverty reduction interventions was directed at a slum or ‘favela’ located just outside the Federal District called Vila Estrutural with 6,500 families (an estimated 25,000 inhabitants).The expected results to achieve this objective were measured by the following three indicators: “(i) improved water quality of the Vicente Pires and Descoberto Rivers; (ii) improved living conditions in targeted settlements by providing integrated urban environmental services; and (iii) strengthened technical and institutional capacity contributing to the sustainable development of the Brasilia metropolitan region (ICR, p.15).

The water quality of the Vicente Pires and Descoberto Rivers has not improved, although that should happen once the wastewater system is fully operational (expected in early 2014). However, risks to water quality of those rivers continue to persist due to the failure to close and clean-up the Joquei Clube landfill’s leachate run-off and continued squatter invasions of the area. The project’s activities in Vila Estrutural resulted in an “overall marked improvement of the quality of life” for many inhabitants between 2004 and 2011 in terms of access to basic infrastructure and community services, such as water, sewerage, and solid waste collection, as well as housing. These improved environmental conditions, basic infrastructure, and community services had previously impeded progress to reduce poverty levels for residents. While attribution is not clear, the project probably had a significant role to play in those improved conditions.

Finally, progress toward strengthening the technical and institutional capacity contributing to the poverty reduction strategies of the region was not demonstrated. Inter-institutional cooperation and coordination was difficult to achieve, and political rivalries between different governmental bodies and complaints from local communities impeded project implementation to the point whereby many project activities and plans had be cancelled or significantly scaled back.

c) Rehabilitating the environment around two key river basins. Rating: Negligible.

The results of the BES Project’s six-year intervention in terms of environmental rehabilitation was to be measured by one of the three project outcome indicators: improvement of the water quality of the Vicente Pires and Descoberto Rivers achieving the class 2 water quality index (IQA) standard, allowing human consumption after conventional treatment. This outcome indicator was later revised to include just maintaining or improving the water quality of the Vicente Pires River after difficulties arose with the land acquisition for the counterpart-funded construction of pumping stations and sludge treatment plant works in Aguas Lindas. However, it should be noted again that revision of this outcome indicator occurred with only one year left to implement the project.

The only information on water quality parameters for either the Vicente Pires or Descoberto rivers was provided in supplemental information subsequently provided by the project team to IEG. The last water quality sampling report available showed only a marginal improvement in water quality for Lake Paranoa, which is the receiving water body for the Vicente Pires River, based on samples taken from July to December of 2011 by ADASA (the Regulatory Water and Sanitation Authority for the Federal District). No definitive information was provided on the “new” estimated or actual completion date for that facility. Since the sewage treatment systems have not been completed yet, the contamination of two very important water bodies (both in terms of their recreational value for the over 200,000 residents of nearby favelas and their role as sources of water supply) in the metro Brasilia area – the Paranoá and Descoberto lakes, has continued unabated. In addition, whatever synergies or co-benefits had been expected to accrue to the BES Project from the GEF-funded Cerrado biodiversity conservation project did not materialize since that project was never approved or implemented by the GEF.

5. Efficiency:

The ex-post economic internal rate of return (EIRR) was reported to be “much lower” (ICR, p. 17) than projected due to the fact that “Water Source Preservation” was the primary driver of net benefits, although how much lower than the estimated ex-ante EIRR of 21.3 was not specified. These economic benefits were anticipated to come primarily from the removal of sewerage discharges from Vila Estrutural and Vicente Pires (partially completed); drainage run-off from Vila Estrutural (partially completed); export and secondary treatment of sewerage from Aguas Lindas (cancelled); and the deactivation and environmental recuperation of the Joquei Clube sanitary landfill (cancelled). None of these expected benefits were fully achieved, and half of the outputs intended to achieve the objective of ensuring the quality and future integrity of water resources were cancelled.

In addition, a financial analysis was conducted with the objective of assessing project investments in terms of their viability and to determine the need for subsidies for those components with less than full cost recovery as required for operation and maintenance (O&M) over the project horizon of 20 years. On all counts, none of the estimated ex-post values were achieved: revenues dropped from US$188.9M to US$59.5M, net revenues (that is, revenues minus capital and O&M costs) dropped from US$180.8M to US$65.4M, and cost recovery percentages dropped from 63.4% to 47.1% while actual costs were three times higher than expected (ICR, pp. 16-18).

Operational efficiency was negligible due to multiple cancellations and delayed construction of major infrastructure works (e.g., the sewage treatment plants and pumping stations, and the Jocquei Clube landfill) and the sudden and unexpected changes in the lead implementing agency by the Borrower without consulting the Bank, which led to the suspension of Bank financial disbursements for over a year. Therefore, overall Efficiency for the project is rated as negligible.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
21.3%
89%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The Relevance of Objectives was assessed as substantial due to the strong connections between the objectives of the project and the Country Partnership Strategy agreed to by the Borrower and the Bank throughout implementation and in effect at the close of the project. The Relevance of Design was assessed as modest due to the overly complex implementation structure and thematic over-reach to address both wastewater treatment and solid waste management issues. This design was approved despite the presence of well-known, but underestimated, political and institutional risks; and inadequate mitigation measures or alternative strategies and mechanisms developed to ensure the continuity of project activities under unstable political conditions.

The Efficacy of ensuring the quality of key water resources in the region was modest due to continuing delays in making the wastewater collection and treatment system operational. Although substantial progress was made in terms of improving the living conditions of local residents, overall poverty reduction strategies were only modestly successful in terms of building the technical and institutional capacity of local governmental bodies and ensuring improved water quality servicing those areas. Progress toward rehabilitating the environment around two key river basins (the Descoberto and Vincent Pires rivers) was negligible since the civil engineering works were not completed and contamination of those water bodies continues to persist in the absence of adequate Government controls.

Finally, the Efficiency of project implementation was rated as having been negligible due to numerous operational inefficiencies and delays, and quantitative measures of economic and financial rates of return that, while unspecified in the ICR, did not attain expected levels.

a. Outcome Rating: Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

There are a number of technical, institutional, and political reasons why the risk to development outcome for the BES Project is rated as high. First, the Federal District Government is still not effectively carrying out land use controls in Vila Estrutural, which continues to experience an increasing number of illegal invasions by squatters. Without concerted efforts to limit further invasions, it is likely that access to urban services, the continued operation and maintenance of infrastructure works, and the overall quality of life will be negatively impacted. In addition, the invasion of environmentally protected lands (and re-occupation of areas that were vacated by households relocated to the resettlement housing) will continue to put water resources at risk.

Secondly, the solid waste management system continues to be plagued by problems threatening its proper functioning and sustainability. Feasibility studies indicating that the new landfill planned to replace the Jocquei Clube dumpsite via a concession to the private sector was a viable option have been proven wrong and grossly underestimated the institutional and legal review hurdles still to be resolved.

More positively, the Borrower, in a letter from the Governor to the Bank, committed to continue implementing the remaining project activities and assume the financial responsibility for payment for any activities carried out after the closing date. Some of these activities included the preparation of the design for the new sanitary landfill, a management plan for protected areas in Vila Estrutural, the 2nd stage of a road and drainage system in Vila Estrutural (all now complete), and a sewage system in Vicente Pires. Funding for the PMU and to supervise the termination of the last remaining resettlement cases was included in the budget for 2012.

However, a number of other activities are still “not under execution” by the counterpart, including: the preparation and implementation of a strategic plan for ADASA (cancelled), the construction of an elementary school in Vila Estrutural (the last one of five planned schools to be built there), the construction and equipping of a solid waste triage center, and a training program for scavenger workers at the Jocquei Clube dumpsite (ICR, p. 14). In addition, a stronger or more coherent institutional framework or coordinating mechanism has not been firmly established or operationalized. The Jocquei Clube dumpsite is still being used while no concrete plans to actually build and operate the new sanitary landfill are imminent. Sewage treatment or urban run-off has not been controlled from contaminating nearby water resources. New invasions of illegal squatters have not been prevented in previously cleared areas. Despite the investment of six years and nearly US$116 million, the downward trend in social and environmental conditions in the metropolitan area of Brasilia does not appear to have been significantly reversed or abated.

a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:
The quality of the Bank’s performance at entry is rated as unsatisfactory for three principal reasons. First, despite a technically sound diagnosis of the most pressing problems facing the larger metropolitan area of the Federal District as part of project preparation, and the development of a comprehensive approach to address them with a strategic focus on regional water resources and the main threats to them, the more detailed institutional and political “nuts and bolts” challenges of implementing this broader vision were not well thought through. The project had a flat organizational structure with 10 Brazilian agencies involved in its implementation, but with none of them possessing the necessary authority or influence over the others to provide the executive leadership needed to maneuver through all of the legal and political issues encountered. These risks had been clearly recognized by the Bank team in preparing and appraising the project’s risks (PAD, p. 10).

Second, the legal and political difficulties of upgrading Vila Estrutural’s physical infrastructure were grossly underestimated, although the project team at appraisal had clearly identified the uncertainties and challenges of obtaining the necessary environmental licenses to construct and operate a new landfill, sewage and sludge treatment plants, and wastewater pumping stations. The Public Prosecutor, Attorney General, and the Court of the Federal District were required to become involved in the project in order to embargo project works that had begun without first obtaining the proper operating licenses. Finally, greater insulation from political factors should have been built into the project, as suggested in the Borrower’s comments. These difficulties revealed the inadequacy of the mitigation measures designed to minimize disruptions to project implementation.

Third, inconsistencies and differences in the institutional and legal framework for resettlement issues between the Bank’s applicable policy (OP 4.12) and Brazil’s own resettlement policies were not adequately considered and harmonized during project preparation and appraisal. The project’s Resettlement Action Plan (dated May 2008) envisioned that: “all affected households would be resettled to new houses (compensation) before any works or clearing occurred at their previous dwelling, thereby avoiding any need for temporary resettlement or payment of rent.” However, this sequencing did not occur. The absence of a consistent resettlement policy and best practices in handling grievances filed by affected households between the Bank and Brazilian counterpart agencies was not foreseen by the project’s staff at appraisal. Its subsequent impact on resettlement plans was dramatic, being largely responsible for the suspension of the Bank’s disbursements to the project for nearly a year and a half from May 2010 until their full reinstatement at the end of September 2011, along with the unconsulted change in Implementing Agency.

Quality-at-Entry Rating: Unsatisfactory

b. Quality of supervision:
Despite some of the very serious problems associated with the design of the project and decisions made at entry, the Bank’s performance during implementation appears to have been satisfactory. This was largely due to its conscientious support of the actions and decisions on the part of Brazilian implementing partners (to the extent possible under the Bank’s own operational policies and legal covenants), and to the additional attention, effort and resources it put into the project once problems with the institutional stability of implementing partners and resettlement issues arose. Sometimes, circumstances made it impossible for the Bank to acquiesce to unilateral decisions by implementing partners without prior consultation (i.e., changing the lead implementing agency twice during the project) or when Bank protocols regarding resettlement issues were not followed. In those instances, the Bank took the difficult, but correct, decision to suspend disbursements, but under very clear conditions for re-establishing them again once those conditions were met.

The Borrower contested this point in their comments, but offered no supporting evidence to substantiate their claim that there was a need for “greater guidance and clarity regarding the rules and procedures of the Bank in the suspension of disbursement, during which conditions were changed throughout the period of suspension” (ICR, p. 51) or that the “extension period granted to the Program—from March to December 2011— was largely undermined by the constraints imposed, hindering the work of the PMU/ADASA” (ICR, p. 46).

Early in project implementation, a gap developed between what was stated in the Resettlement Action Plan and what was being implemented on the ground, especially with regard to temporary resettlement. The Bank team worked with the implementing agency to prepare time-bound actions to resolve these issues. Community engagement and communications were strengthened, Bank supervision was significantly increased to about 4-6 supervision missions per year intensified with increased budget, and the project was put on a management watch list. The Bank’s project team continued to provide implementation support to the Government to achieve the project objectives and comply with resettlement and safeguards obligations, which remained in place for six months (later amended to one year) after the closing date of the loan. Senior management and leading Bank experts in social development and resettlement safeguards (including the Chief Counsel for safeguards, and the Sector Manager for Social Development) participated in key missions. Thus, finding no fault on its part, IEG rates the Bank’s quality of supervision as satisfactory, reflecting what was seemingly an exemplary effort made by the Bank project team under very difficult circumstances.

With Quality at Entry in the unsatisfactory range and Quality of Supervision in the satisfactory range, the outcome rating is the determining factor, resulting in an overall Bank Performance rating of Moderately Unsatisfactory.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The Government’s performance is assessed as having been unsatisfactory due to a number of factors, among them: (i) failing to abide by contractual obligations with regard to prior notification of the Bank in changing implementing agencies; (ii) not adhering to Bank resettlement safeguards (OP 4.12) or resolving outstanding resettlement grievances brought by affected resettled parties; (iii) not acting forcefully enough to ensure greater inter-agency coordination and allowing internal conflicting political agendas to interfere with agreed upon project achievements and results; and (iv) exacerbating risks related to the multi-agency model adopted by the BES Project that resulted in unnecessary delays and a lack of coordinated actions.

An initial change of the lead implementing agency occurred within the first year of project effectiveness in April 2007 when the Government decided to assign responsibility for the overall coordination and implementation of the program from the Water, Energy and Sanitation Regulatory Agency (ADASA) to the Secretary of State for Urban Development and Environment of the Federal District (SEDUMA). This revoked a previous decree of May 2005 delegating authority to ADASA. Then, in March 2010, the Government reversed course again by transferring project leadership and inter-agency coordination responsibilities from SEDUMA back to ADASA. This created legal ambiguity in implementation, especially regarding fiduciary responsibilities, requiring the project to be restructured and the loan agreement to be amended. In addition, once a new government was established in 2011, all staff in the secretariats dealing with resettlement issues were changed, leading to a transition period and a re-learning process at a critical stage of project implementation.

To its credit, the Government did assume financial responsibility for continuing a number of project activities not completed by the closing date, such as strengthening the environmental licensing procedures of IBRAM, preparing the design for the new sanitary landfill, developing a management plan for protected areas in Vila Estrutural, designing the 2nd stage of a road and drainage system in Vila Estrutural as well as the sewage system in Vicente Pires (all now complete).

Government Performance Rating: Unsatisfactory

b. Implementing Agency Performance:
The performance of the various implementing agencies was unsatisfactory on a number of counts. First, as noted before, the ICR made several references to the recurrent resettlement issues that the project had during implementation, due largely to inconsistencies between the Bank’s and implementing agencies’ resettlement policies and practices. Five different secretariats were responsible for various aspects of the resettlement program. Initially, the Project Management Unit (PMU), under the leadership of SEDUMA and ADASA, only had a relationship with SEDUMA, which did not have the institutional power to influence or even monitor the work of the other four secretariats (despite formal agreements signed between the entities to implement the project).

The disbursement of project funds was suspended on May 7, 2010 due to non-compliance with the Bank’s Safeguards policy regarding resettlement (OP 4.12), which had been flagged more than a year earlier in April 2009, and by the unconsulted change in Implementing Agency (from SEDUMA back to ADASA). The suspension was lifted when houses for those displaced by the project were constructed and delivered, and the Bank’s assessment of ADASA’s capacity to implement the project had been completed and found satisfactory. The ICR (p.24) noted that, “Data provided in status reports were also on various occasions inconsistent with previous reports or incomplete.”

Implementing Agency Performance Rating: Unsatisfactory

Overall Borrower Performance Rating: Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The ICR states that the project M&E system required three sets of information: (a) water quality data set for the rivers benefiting from project interventions; (b) an integrated social-environmental assessment of the urban living conditions in Vila Estrutural and Aguas Lindas; and (c) monitoring and reporting of project outputs and results achieved per activity or component (p. 10). Thus, the indicators selected reflected the project’s objectives and indicators of the extent to which improvements in environmental conditions and the quality-of-life of targeted communities was being achieved by project interventions. The indicators selected were measureable and meaningful although the Social Environmental Indicator (ISA-F) turned out to be too complicated and data- intensive to replicate, and was re-focused after the Mid-Term Review. A consultancy firm with expertise in managing internationally financed development projects was contracted to assist the PMU perform specialized tasks to manage multi-sectoral project activities, including M&E. However, as a result of “outsourcing” these functions, the M&E system may have lacked sufficient stakeholder participation and ownership.

b. M&E Implementation:
The water quality data for the river basins targeted by the project was obtained from CAESB, which already monitored the Federal District’s river water quality through a permanently staffed monitoring program. CAESB provided timely and reliable information on water quality to the PMU, and its existing ‘Water Resource Master Plan’ provided the comparative baseline for measuring progress during implementation and as part of the ex-post assessment. This water quality monitoring program continues functioning to this day.

A Social Environmental Indicator (ISA-F) was used to monitor improvements in the quality-of-life for residents living in Villa Estrutural (note: Aguas Lindas was dropped from the project design as part of the first project restructuring). However, prior to the Mid-term Review, it was determined that using the ISA-F was too complicated and data-intensive to be implemented. Therefore, the PMU extracted 46 indicators (this was later reduced to 14 indicators) from the larger set of ISA-F criteria to assess the quality-of-life in Villa Estrutural. Thus some weaknesses in the M&E design were later remedied during implementation.

Finally, with regard to the third set of information collected by the M&E system, the project’s activities and outputs were obtained through “regular project implementation monitoring” carried out by the PMU. The M&E system also included contracting a consultancy firm with expertise in managing M&E systems to assist the PMU. The consulting firm performed all tasks required to track the outputs and results achieved through project activities.

a. M&E Utilization:
No information was provided in the ICR regarding how the M&E data was used by project managers to adjust or change the project’s activities or course. Therefore, while the M&E system was implemented according to the Results Framework and M&E system design (with the obvious exception of the downscaled ISA-F mechanism), there is a lack of information in the ICR regarding whether it had any impact on project execution or decision-making processes, or on subsequent interventions following the closure of this project.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:
Three World Bank safeguard policies were triggered by the BES Project: OP 4.01: Environmental Assessment, OP 4.04 Natural Habitats, and OP 4.12: Involuntary Resettlement. The project was classified as an “A” category project, meaning that it had a full environmental assessment conducted due to concerns about the potential environmental impacts of project interventions in the two targeted river basins: the Vincente Pires and Descoberto.
With respect to the comprehensive Environmental Assessment that was carried out in accordance with Bank policy for all Category “A” projects, the mitigation and enhancement measures, including associated costs, responsibilities and schedule, to address anticipated adverse environmental impacts were identified and summarized in the Environmental Management Plan (EMP). The Project was expected to have a highly positive impact on biodiversity and natural habitats. There was no mention made in the ICR regarding whether the project actually implemented the mitigation measures set forth in the EMP or made any modifications to it when the project was restructured at the end of 2010. Nor was there any mention made regarding whether the applicable safeguard policy for Environmental Assessment (OP 4.01) was conducted independently.

Whereas there do not appear to have been any non-compliance issues with the other two safeguard policies triggered by the project (OP 4.01 and OP 4.04), according to the ICR, the Bank started receiving complaints from community members regarding resettlement actions taken by the Government beginning in August 2008. Compliance with the Bank’s resettlement safeguard policy (OP 4.12) was problematic throughout project implementation, resulting in many delays in rolling out infrastructure works and resettling people into new, adequate housing served by basic public services. Specifically, there were concerns about households being temporarily resettled into tents and wooden shacks—both of which were not options included in the Resettlement Action Plan (May 2008).

b. Fiduciary Compliance:
Fiduciary compliance remained satisfactory throughout project implementation. Despite some initial problems, the project’s financial management system functioned efficiently and reliably in providing timely, accurate information to manage and monitor project implementation. However, the ICR (p.29) notes that three independent audits were conducted, but the final external audit was not carried out until after the closing date of the project. It did not indicate whether the auditors’ opinions were qualified or not, and if so, whether any corrective measures were taken to address them.
In terms of procurement, continual problems with procurement processes were encountered by the project, resulting in multiple delays in completing an agreement on a concession deal to build and operate the new sanitary landfill and close the Joquei Clube dumpsite. Similar failures in procurement were encountered in implementing a recycling program for Government offices, constructing a solid waste triage center, and training scavengers at the Joquei Clube dumpsite, none of which were completed by the project’s closing. Disbursement of project funds was suspended for five months between May and October of 2010 and was not fully lifted for another year (until the end of September 2011) due to the aforementioned non-compliance issues with the Bank’s resettlement requirements and unauthorized change in the Implementing Agency. Throughout implementation, counterpart funding seriously lagged disbursements of the Bank loan even though the project was supposed to be financed equally by the Bank loan and counterpart funds (50-50). The latter disbursement was three times slower (ICR, p. 23).

c. Unintended Impacts (positive or negative):
None.

d. Other:
No other substantive issues not mentioned elsewhere.



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Unsatisfactory
Unsatisfactory
 
Risk to Development Outcome:
Significant
High
Failure to control continuing invasions of unsafe or unhealthy areas within Federal District slums along with incomplete WS&S and SWM works warrants the higher risk rating. 
Bank Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Borrower Performance:
Unsatisfactory
Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
Urban upgrading projects are best managed by one agency with overall responsibility and sufficient delegated authority over other collaborating agencies to carry out all activities under the project. This lesson was adapted from the ICR. The primary implementing agency requires inputs from multiple municipal agencies (i.e., water and sanitation, transport and other public works, housing, education, and recreational/social outlets), but also needs to have strong central leadership institutionally with sufficient authority to effectively coordinate the proper sequencing and pace of implementation activities and civil works projects.

An accurate and realistic assessment of the political environment and political agendas of all key stakeholders should be a key activity in the preparation of complex, integrated development projects along with back-up contingency plans for minimizing potential politically motivated resistance or institutional inaction. The BES Project had a holistic design, but it was premised on an overly optimistic assessment of the collaborating institution’s collaborative capacity. It failed to account for their separate and conflicting political agendas. This point was clearly made in an ex-post evaluation of the project by 19 representatives from many of the implementing agencies who cited the need to minimize the “perverse incentives” created by having so many politically-motivated decisions and actions being made by constantly changing institutions without clear leadership and coordination authority assigned. It is important that projects anticipate the in-fighting that is almost certain to occur, and prepare for these rivalries within and between institutions. One cannot assume that just because the greater good is being served by a project that individual pockets of resistance within the Government, private sector, general public, or even among small numbers of affected parties will not form if they do not achieve their interests or if they see an opportunity for further gains through resistance.

In project design the inherent tension between maintaining the simplicity of a project design versus taking an integrated, comprehensive approach to address the different aspects of a development challenge should be explicitly weighed. It is always easy to fault a project for having been too ambitious in its thematic scope or geographic reach, but it should not be the Bank’s goal to do only what is easy or safe. It is difficult for project designers to resist the temptation to want to address all elements of a problem. However, it is important to consider the costs that come with incremental increases in complexity, the coordination required between partners, and the value of the links between the thematic areas. In this case, it might have made more sense to have left the solid waste management issues associated with containing residual groundwater contamination from the Jocquei Clube dumpsite for another project. Handling just the water supply and wastewater treatment issues of the target communities to limit their public health risks and environmental damages to the two specified river basins might have been more manageable and achievable.

Greater attention needs to be given to the issues most likely to pose the greatest risks or difficulties for implementation, such as resettlement. In this case, more emphasis should have been placed on ensuring the consistency and compatibility of the Bank’s and Borrower’s resettlement policies and practices. Papering over differences on sensitive issues did not serve the best interests of either party. By allowing vague, inconsistent, and incomplete processes to remain in project implementation agreements, difficult problems were left to be untangled later that might have been ironed out in negotiations.

14. Assessment Recommended?

Yes
Why?
Given the high aspirations attached to this project initially, followed by its failure to deliver on its promises and expected results, a closer examination of the reasons for the collapse and underperformance would enhance learning. Urban upgrading projects such as this one are likely to become increasingly important responses to pressing social, environmental and political demands in the future.

15. Comments on Quality of ICR:

The ICR was clearly written, covered all of the required areas for a project self-assessment, was factual and logical in making its arguments, and made reasonable and realistic assessments of the BES Project’s performance. Several sections of the ICR were exceptionally well-written and provided powerful insights into the project’s failure. These included the in-depth treatment of resettlement issues and on-going procurement problems, and the Lessons Learned section. In addition, the ICR clearly laid out changes in the project’s structural and thematic orientation following its restructuring in 2010 with a table presenting a side-by-side comparison of the original and revised outcome indicators. However, a few sections of the report on Safeguard implementation and M&E, lacked sufficient depth. There were also some inconsistencies in reporting the status of various project interventions in the report, such as whether the Jocquei Clube landfill procurement process was cancelled or merely revised. There was no mention made regarding whether an independent review of safeguard implementation issues was conducted or not, or whether the project actually implemented the mitigation measures set forth in the Environmental Management Plan. Otherwise, it was a well-organized and well-written report.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions