Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Recife Urban Upgrading Project


  
1. Project Data:   
ICR Review Date Posted:
12/23/2013   
Country:
Brazil
PROJ ID:
P049265
Appraisal
Actual
Project Name:
Recife Urban Upgrading Project
Project Costs(US $M)
 84  114.4
L/C Number:
L4690
Loan/Credit (US $M)
 46  46
Sector Board:
Urban Development
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  04/24/2003
 
 
Closing Date
03/31/2009 03/02/2011
Sector(s):
Water supply (30%), Solid waste management (30%), Housing construction (30%), Sub-national government administration (5%), Central government administration (5%)
Theme(s):
Urban services and housing for the poor (40% - P) Land administration and management (20% - S) Environmental policies and institutions (20% - S) Participation and civic engagement (20% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Dale M. Hill
Peter Nigel Freeman Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The project development objective (PDO) in the appraisal report (p 2) and the ICR is:"to improve the wealth and well being of the urban poor of the Beberibe River Basin in the Recife Metropolitan Region (RMR) by increasing the institutional capacity of public (state and local) and civic entities to plan for, deliver and maintain basic shelter and urban services for the low-income population in a coordinated and sustainable manner."

The PDO in the legal agreement (p 20) uses more precise legal language, and is thus more monitorable, and clearer on accountability: (differences italicized): “to improve the wealth and well being of the urban poor in the RMR [Recife metropolitan Area] by increasing the institutional capacity of public (at the state and municipal level) civic entities to plan for, deliver and maintain basic shelter and urban services for the RMR’s low-income population in a coordinated and sustainable manner."

IEG chooses the formulation in the legal agreement since it is the legally binding document. After restructuring, the objective was not changed, but the scope of the project and the number of beneficiaries did change, although formal documentation did not reflect changes in number of beneficiaries, (TTL meeting).

Articulation of the objective: IEG notes that the phrasing of the objective explicitly mentions building institutional capacity of the State and municipal Governments, as a means to achieve results (and sustain them). For purposes of efficacy, IEG considers this a sub-objective.
.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
The project in Portuguese is known as "Prometropole I". A follow-on project under preparation is referred to as "Prometropole II". The ICR notes (p. 3 footnote) that project components in the PAD and legal agreement differ; the ICR thus uses the description in the legal agreement, as does IEG's review. The figures given in parentheses below are: a) original estimated costs at appraisal; b) revised estimated costs after 4th restructuring amendment; c) actual estimated costs at closing; all three include Government (state and municipality) counterpart funding, and are denominated in millions ("m"). Costs at closing are assumed to be "estimated" since some project activities are continuing after loan closing, with additional Government counterpart funding, and WB post-closing supervision missions.

Part A: ($11.8m; 6.5m; 5.4m) Beberibe River Basin Investments: Carrying out of infrastructure and environmental projects for the overall improvement of the built and natural environment of the Beberibe River Basin, consisting of: (a) creation or rehabilitation of parks, including the ecological park along the margins of the Beberibe River, pocket parks within each of the Sanitation Collection Units (UEs – Unidades de Esgotamento Sanitário) and the Matadouro de Peixinhos complex of historic buildings; (b) construction or rehabilitation of roads and accesses, including improvements in the general flow of traffic; (c) improvements in macro drainage to reduce flooding; and
(d) improvements in the transport systems of sewage to be treated by the Peixinhos waste water treatment plant.

Part B: ($5.5m; $8.3m; $5.4m) Water Distribution: (a) Installation of pipes and carrying out of the associated works to improve water supply and distribution within the UEs in the Beberibe River Basin; (b) Acquisition of micro meters for household installation in the UEs; (c) Installation of macro meters and the carrying out of associated works required to isolate the water distribution network within the UEs.

Part C: ($52.1m; $85.2m; $83.7m) Investments in Low-Income Areas: Carrying out of urbanization projects in groups of low-income settlements located in the UEs 03 (R), 04 (R), 08 (R), 17 (R), 19 (R), 20 (R), 21 (R) and 23 (R), which are in Recife, and 07 (O), 12 (O), 13 (O), 15 (O) and 17 (O), which are in Olinda, consisting of water supply and sanitation improvements and household connections, secondary road improvements, micro drainage works, installation of public lighting, creation or rehabilitation of public parks and open spaces, resettlement of population from risk prone areas, and other related investments.

Part D: ($10.5m; $6.9m; $5.3m) Complementary Urbanization Activities: Carrying out of activities complementary to the activities under the preceding parts of the Project, consisting of: (a) engineering design services for the sub-projects; (b) development of social outreach and community participation activities; (c) provision of environmental education to the communities; (d) feasibility studies for a sites and services project; (e) studies for expanding to other areas in the RMR the investments made under the Project; (f) Project monitoring and evaluation activities; (g) a feasibility study for the establishment and operation of a micro credit program for housing improvement and small business development in the Beberibe River Basin (and provision of technical assistance for the operation of such micro credit program)(this component dropped in restructuring); and (h) formulation of the Environmental Management Plan (EMP).

Part E: ($4.1m; $13.0m; $14.6m) Project Management: Provision of technical assistance to Pernambuco State Planning and Research Agency (CONDEPE/FIDEM) and the Municipalities in the management, implementation and coordination of the Project.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost:
The Project’s scope needed to be tightened during implementation, due to both: (a) significant increase in costs (55 percent appreciation of the Brazilian Real against the US Dollar, and 132 percent increase in construction sector costs in Brazil); and b) 37 percent increase in the number of families to be resettled. (See the Safeguard and Fiduciary Compliance Section for more detail) (ICR, p.4). In addition, the Water and Sanitation Company COMPESA raised its standards, making the cost estimates for that component at appraisal too low (ICR, p.7). Broadly, the Project’s components were revised as follows: (See ICR Table on p. 5 and Annex 10 for more detail): Parts of Component A were financed under Part C; Parts of C (benefitting both municipalities of Recife and Olinda) were removed from the WB-financed portion of the project, since Government's newly launched "Growth Acceleration Program (PAC)" which financed housing and infrastructure, allocated some finances to the State of Pernambuco to complete the works; and Part D which was to include micro-development and small business development was removed, in favor of completing engineering designs for the major works.
Financing:
The PAD states (p. 22) that the municipalities of Recife and Olinda were to contribute $20m in counterpart funding, and "the rest" (IEG calculation works out to $18m) would come from the State of Pernambuco. The Legal Agreements do not mention specific amounts of counterpart financing. Initially, due to the increase in inflation, and the resultant increase in cost estimates, the scope of the project was cut back. Later, the Government(s) increased counterpart funding by $36m, in order to ensure completion of the agreed components. (ICR, p. 7). Obtaining approvals for this financing took some time, however, and some components were not completed at closing. The Government(s) committed to complete the works, and WB post-closing supervision continues on those components.
Borrower Contribution:
The ICR is unclear whether the original expected counterpart funds were provided on time ($20m plus 18m). Reference is made in the ICR (p.6) to an implied delay ("another risk that was underestimated was the delay in counterpart funding"), but the time frame is not clear--whether that was a delay in provision of the original counterpart funding, or the later pledge. The ICR reports in a later section (p.7) that "securing ...funding at the federal level" complementary to the additional $36m funded by the state and municipal government "further delayed the Project's progress." It is also not clear if the full committed counterpart funding is reflected in the final project cost estimates reported in the ICR, or if only the part disbursed at project closing is included.
Dates:
The project was appraised in October 2001, approved in April 2003, and became effective in October 2003. (ICR, p.i)
Restructurings (not subject to Board approval, but sometimes requiring legal amendments) took place at the following dates (ICR, p. viii):
01/23/2004 (Disbursements at this date were $0.9m). (To create a second disbursement category for goods and expand the Project description for Part E, project management);
10/13/2006 (Disbursements at this date were $2.9m). (To create an additional subcategory for works and reallocate loan proceeds among disbursement categories);
03/06/2009 (Disbursements at this date were $29.0m). (To include engineering designs for waste water collection micro basins, incorporate electronic bidding among the procurement methods acceptable to the Bank, reallocate loan proceeds and disbursement percentages among categories, and extend the closing date for a period of 18 months);
06/10/2010 (Disbursements at this date were $39.1m). (To extend the closing date by five months, to reallocate loan proceeds and change the disbursement percentages and to revise the Project's scope).
The original closing date was 3/31, 2009; the Project was extended for a total of 23 months due to "design-related delays, political impediments, unanticipated cost escalations, and a dramatic increase in the number of families to be resettled." (ICR, p.1). The final closing date was 03/02/2011. As noted, work on some components continued past the closing date, with Government financing, and WB supervision support financed from budgeting for preparation of follow-on projects. (TTL meeting)


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High.
Problems and Consequences: In 2000, over 12 million persons or 3.2 million households lived in precarious housing within slums ("favelas") in Brazil. As the Project Appraisal Report (PAD) noted (p. 5-9), living conditions in low-income areas in Recife Metropolitical Region in the Northeast (RMR, home to 1.2 million) were recognized as particularly poor, demonstrating a high need for improvements to upgrade housing safety, regularize development patterns, provide urban services and infrastructure and alternatives to illegal connections, and to increase land security. The low income settlements were haphazard, and often located in environmentally sensitive and high risk areas. Poor drainage, lack of sanitary services and uneven solid waste collection caused health risks. The project focused mainly on the Beberibe River Basin, where 66% of the 550,000 residents lived on less than two minimum salaries.

Relevance at Time of Appraisal: Before the project, the Government of the State of Pernambuco (GSP) led a number of initiatives to manage interventions in low-income urban areas, but the programs were small-scale, did not approach interventions in an integrated manner, and had a limited impact. As Recife and Olinda became increasingly urbanized and informal settlements even more common, the municipalities wanted to develop a framework to systemize urban upgrading and land management, and called on the State of Pernambuco to request assistance from the Bank (ICR, p.1).The project was relevant at the time of appraisal, being consistent with the WBG Country Assistance Strategy (CAS), and responsive to the State's call for assistance.

Continued Relevance: Government Priority The need is still there today, and the objective is still relevant. New demand for housing in Brazil is forecast to reach 23 million units by 2023, with about one third of the housing deficit concentrated in Brazil‘s 11 largest metropolitan areas (RMR is the fourth largest). Showing the Federal Government's priority to address these issues, during 2011-2014 it plans to invest $19 billion in PAC Favelas, the world‘s largest slum upgrading program, and almost $45 billion in the Minha Casa Minha Vida (MCMV) program of housing subsidies for low and middle income households. The Government of Brazil (GoB) has also announced the multi-sector "Brasil sem Miséria‖ program (BSM)" which targets the 16.2 million people that live on less than R70 per capita per month (about $1.5 per day). 59% of the extreme poor are located in the Northeast.

Continued relevance: WBG Country Partnership Strategy (CPS): Supporting these programs is consistent with the Bank Group’s Country Partnership strategy for Brazil, which emphasizes creating opportunities for reduction of poverty and inequality by targeting the poor and vulnerable, supporting growth and employment, and improving the provision of public services for low income households (among other themes) (CPS, p.21).

b. Relevance of Design:
Modest

Although the objectives did not change after restructuring, the scope, components and number of beneficiaries (and financing) did change, due to inflation, increases in construction costs, increased number of resettlees and increased standards and requirements for the provision of water supply.

Relevance of Original Design relative to objectives:
The original design emphasized integrated urban development and included the following investments in urban upgrading: components for resettlement of population from risk-prone areas, housing improvement, water and sanitation supply, roads, street lighting, rehabilitation of public parks, and improved drainage for flood control. These constitute an appropriate mix of components to improve the well-being of the low income population. The development of public parks was not only to reduce social exclusion, but also to protect environmentally sensitive areas from informal urban encroachment. The sub-objective in respect of institution-building of state and local Governments was not directly linked to the project's investments. Rather, institutional capacity was to be built via hands-on experience. However, the design did not allocate sufficient resources to this activity or sub-objective--only 5% of total costs, (ICR, p. 5-6).

Other reinforcing components, which were important in increasing the flow of benefits from this infrastructure included support for community participation and consultation, provision for studies of a possible micro finance and small business development component, and the installation of water meters for more efficient water use and cost recovery. Indicators related to these components' contributions included a survey of perceptions of quality of life of the beneficiary population, and measures of changes in cost recovery.

While the project further included provision for needed engineering and feasibility studies, an M&E system, and formulation of the environmental action plan, the funds provided for institution-building, an explicit part of the objective, consisted only of project management (component E) and were not sufficient. (<5% of total project costs). The Government found the support for project management helpful, but commented that some structured support on technical issues such as land titling were needed and would have been welcomed. (ICR, p. 5 -6). The ICR stated that the need for technical assistance (TA) support for resettlement had not been anticipated, and was not built into design (p. 18). Arrangements for ensuring appropriate maintenance were not well detailed.

Relevance of Revised Design relative to Objective:
Modest
After the cost escalation and other factors forced several restructurings, the geographical scope of the project was cut back (fewer sanitation collection units - EUs), the study for the microfinance and small business development component was dropped, (ICR, p. 4) and the scope of the technical assistance (TA) aiming to support the issuance of land titles was cut back--land titles were to be issued to project beneficiaries, but not all residents of the area. A minor component related to retaining walls was also dropped (TTL meeting). A follow-on project will provide broader TA to the State and municipalities in the important area of land titling. Positive features of the redesign included: (a) retention of the component providing support for community participation and consultation, and social worker services; and (b) increase of the amount allocated to project management. IEG regards the changes in scope and design changes to have been appropriate, given the changes in prices and standards which created the need for restructuring; however, the problem of insufficient provision for institution-building remained.


4. Achievement of Objectives (Efficacy) :

First Objective: to improve the wealth and well being of the urban poor in the RMR [Recife metropolitan Area]...." Substantial The following achievements reported (from ICR, p. iv) are achievements relative to original targets (no revised targets are included in the tables); IEG has included information relative to revised targets where the narrative covers this.

Outputs Completed:
Water Supply: One reservoir, 96% of water supply systems and 87% of household connections were completed. Plans are underway for completion. Attempts to measure cost recovery improvements are covered under second objective.
Drainage: 92% of contracted meters were completed. Plans are underway for completion.
Increase in length of thoroughfares w/proper lighting: ICR says “met” but indicators given do not support this; plans underway for completion.
Resettlement: 83% resettled; 289 families still to be resettled. Plans underway for completion.
Integration of community-based groups in slum upgrading processes: all groups targeted after scale-back were integrated into processes.
Roads: 96% roads completed; but key “Road Presidente” not completed and said to be affecting commerce (TTL meeting). Plans underway for completion.
Increase in access to recreational activities: 66% completed. Plans underway for completion.
Titling: number of new properly titles issued: 83% met; the remaining in process.
Direct employment of community members in execution of project works: 100% met (but definition unclear).

Outcomes:
Increase in property values: $7,350 increase per property in slum upgrading (exceeds appraisal target of $2,963); $24,278 increase per property in resettlement areas (exceeds target of $3,774). These high values reflect also lower beneficiary coverage after restructuring (5,040 households instead of 35,000; ICR p33, Annex 3) and some effects of growth in demand for housing generally. TTL stated that cited values netted out inflation. (See also section on efficiency).
Improved Quality of Housing (private investment): “ICR says “met”. and statistics reported from a survey in the 6 UEs appear to support this; but lack of explanation on sampling and significance of differences makes this interpretation tentative.
Perceptions of Quality of Life: ICR says “met”; however, sufficient information to justify this judgement is not given, as data are reported for different UEs, with no information on statistical significance. For example, from scanning the data, unambiguous improvement over baseline in all service areas sampled (UEs) are evident only in three sectors of 11: water supply, sewage services and leisure areas. In some service areas, there were declines in perceptions of some services (pavement quality, public transportation, health services.).

While the number of UEs was cut back, and the number of resettlees increased, the objective did not specify the geographical boundaries or the number of beneficiaries targeted, so the changes in components are not considered in the efficacy rating. (The substantial delays are considered in “efficiency”)

Second Objective: "... increasing the institutional capacity of public (at the state and municipal level) civic entities to plan for, deliver and maintain basic shelter and urban services for the RMR’s low-income population in a coordinated and sustainable manner. "
Rating: Modest
Outputs: The planned assignment of responsibility to the State and municipal Governments for execution of components of the project was met: 47% of spending involved components executed by the state (compared to a “target” of 43%), and 51% involved components executed by the municipalities of Olinda and Recife (compared to “target” of 57%. However, for this indicator to properly reflect “institution building,” assumptions need to be made about the quality of the execution, the experience and learning gained, and the ability to apply this to future similar projects. Given the limitations of such an indicator (and no baseline) and considering there were no other outcome indicators that measured the impact of this experience, the ICR relied on the Borrower's final report and anecdotes supplied by the institutions concerned to assess this outcome. In general, the municipal and state entities believed that through the project they enhanced their ability to deliver and maintain basic shelter and urban services for low income people, particularly in the areas of resettlement. In regard to resettlement the GSP and the municipalities developed a standardized approach (that included community participation and social outreach) which they are currently utilizing in the execution of urban upgrading projects. Olinda Municipality described the approach as a "paradigm shift". Nevertheless, there is an absence of hard evaluative that institutional capacity building has been successful.
The labeling of the indicator for "cost recovery" is incorrect, since what is measured is the "benefit" of increased water supply coverage, and many of the new recipients are exempt from payment.
Outputs Not Delivered: Technical assistance in land titling sufficient to issue land titles for all the residents in the target area;
Outcomes Not Realized: Institution-Building: Not explicitly measured and little evidence presented. Notably, the ICR's rating of implementing agencies is moderately unsatisfactory, and note is made of issues in both fiduciary and safeguard management that emerged during implementation, including issues on municipal accounting revealed in the final audit of the project.


5. Efficiency:

Methodology: A consulting firm, Datamétrica, set out to measure two types of benefits: a) increase in housing values in the project area (capturing effects of housing and infrastructure improvements); and b) improved water service coverage, using a proxy of increased revenues of the water supply company in the project area. For comparisons between estimated benefits and costs, total project costs were used, with an assumption of 1% maintenance. IEG's assessment is based on the ICR coverage. (Consultant report was in Portuguese.)

--Housing: The broad approach is sound:-- a) to use increase in housing prices as a proxy for the benefits of improved housing and infrastructure; b) to estimate the probable increase in housing values using regression analysis with the log of housing prices as dependent variable and various features of the housing and community services (some proxies) as independent variables; and c) to compare the resulting price increase estimates to (i) PAD estimates; (ii) costs, and (iii) housing with no improvements. But data sources and dates, and details of the way the approach was actually applied are not fully presented, and some aspects of the econometric analysis can be questioned. Also, separate estimates of benefits and costs were made for the Beberibe Basin, the “low income areas” and the resettlement areas, but the results of each are not reported fully.
--Three major factors which make results hard to interpret are: a) the decrease since restructuring in the number of beneficiaries, particularly in the “low-income areas” (from 35,000 to 5,040); b) the fact reported in the ICR (p. 32-33) that “macroeconomic conditions in Brazil have improved substantially since the PAD (2003)... with positive repercussions in terms of job creation and distribution..... The corollary of this phenomenon is a steep rise in real estate prices on the Brazilian market, sustained by the growth of employment, income and credit. It is possible, then, that the estimated post-program rise in real estate values also reflects the current upward movement of the national real estate market."; c) the ICR also reported an increase in inflation, and specifically in construction costs during the implementation period.
--Results: Housing Prices/Values: For the Beberibe Basin:(a) the estimated post-project housing benefits exceeded those of the PAD both in total and in per household terms: (e.g. $971 vs. $377 per household); (b) the post-project estimated benefits also exceeded project costs in equivalent terms (e.g. $971 vs. $56 per household); (c) properties with characteristics typical of project-financed improvements exceeded estimated benefits of those without those characteristics by 43%.. In contrast, the figures given for the low income areas indicate higher costs than benefits per household (e.g. costs of $9,225 vs. $7,349 benefits per household. Results for the resettles are also presented, but are hard to interpret. Net present value (NPV) was $30m and $34m for discount rates of 12 and 9% respectively.
--Results: Water Service: The benefits of new water service coverage were calculated based on estimates of increased revenues of the water utility, using data from a census survey (since data from the company was not available, ICR, p. iv.) Use of this proxy assumes that charged water rates bear a reasonable relationship to the value that beneficiaries place on the service. Datamétrica took the average water bill, multiplying by number of residences benefitting from new connections (7,202), considering the payment rate of 87%. That methodology resulted in an estimated annual "economic benefit" from water distribution of $1.7m. The NPV of the water services improvement was $4.8m and $7.5m using a discount rate of 12 and 9 percent per annum, respectively.
--Internal Rates of Return (IRR): Separate estimates were calculated for the two components above by comparing benefits to costs and discounting over 20 years. The estimated IRR for housing was 40 percent, 3 percent higher than the target projected in the PAD. The estimated IRR for the water supply was 21.7%, compared to the PAD estimate of 19% . (Coverage of each of these out of project costs was not provided, nor overall IRR estimate.)
--Factors positively affecting efficiency: Government commented in its completion report that hiring a project management company to provide additional support alongside permanent government staff assigned to the project (including training) was an important factor in the successful progress of the work, as was the ongoing participation of the communities in the discussions. (ICR, p. 41)
--Factors negatively affecting efficiency: (a) In the early years of the project, elections resulted in political misalignments, which was said to slow down project implementation; (b) The unanticipated currency depreciation, inflation, construction cost increases, and increased water supply standards led to the restructurings, with attendant contract renegotiations, causing delays and increasing costs; (c) The reluctance of the Government to enforce the cutoff date for registering eligible resettlees led to increased demand which delayed resettlement (at the time of closing, some of these are still in rental housing subsidized by the Government.); (d) lack of optimal provision for support on resettlement and other technical assistance likely reduced project efficiency. The compounded effect of all these factors, summed on ICR p.1, was that the Project was extended for a total of 23 months, and indeed, work on some components continues past the closing date, through Government financing, and continued WB supervision support. (Note this means that some delays and costs are not factored into the economic analysis).
Rating: Modest

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The relevance of the objectives was high and that of design was modest. The ratings of efficacy for the two sub-objectives were substantial for improving the wealth and well being of the urban poor, but modest for building institutional capacity. IEG weights the first objective as more important however. Efficiency is considered modest. Overall IEG assesses the outcome as moderately satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The ICR (p.17) identified risks as “moderate” at time of evaluation, while mentioning some unanticipated risks: (a) the institutional risk (municipalities and FIDEM unable to complete the project and ensure maintenance) (b) technical risk (inability to do feasibility studies in areas where people are living in crowded conditions); and (c) economic risks—affecting ability to stay within 15% of cost estimates. The project also proved vulnerable to: (d) political risks, specifically election cycles. In 2004, just after Project effectiveness, municipal elections were held, resulting in a political disconnect between the Mayors and the Governor, which was not resolved until 2008. All these risks will continue as the project entities try to complete execution of the project with Government funding, and will be relevant also to the Government’s next ambitious scale-up effort. In addition, IEG identifies the following risks to development outcome: (e) financial: risk that housing investments and water systems will not be maintained; (COMPESA’s viability is said to depend on negotiation of a public private partnership (PPP)); and( f) social and environmental risk –possibility of continued “favelization” as the poor continue to move into designated areas of development, straining both environmental resources and public services’ ability to deliver.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:
Positive: In addition to conducting a detailed analysis of the living conditions in the Beberibe River Basin, the team analyzed related projects in the country and same sector and used the knowledge gained from these past operations to develop three alternative structures for the Project that ranged from integrated interventions to sector specific investments. The Bank then selected the structure of the Project based on, among other criteria, the number of poor potentially reached, the visibility and urban impact, the sustainability of the investments, the impact on habitability and risk reduction, and the ability to replicate the Project. (ICR, p.5, 18). The design was appropriate to meet the Project’s objective -–it targeted the poorest neighborhoods in the RMR, integrated urban housing and water and sanitation (WSS), and incorporated participatory components that were likely to meet beneficiaries’ needs. Government mentioned in its ICR its appreciation (ICR, p. 41): "The Bank, in addition to serving as the program’s funding agent, definitely contributed to the success of Prometrópole by sharing its expertise in improving quality of life for populations living in extreme poverty…."


Shortcomings: The central weakness of the background analysis was the lack of a comprehensive assessment of the institutional capacities of the implementing agencies, and inclusion of appropriate components and targets related to this objective. Less than 5% of the original project costs were allocated to project management and institutional support. The ICR states that, considering this gap, the target of improving the quality of life of 150,000 low-income people living in the Basin was overly ambitious for the Project’s scope and financial resources. (ICR, P.5). Institutional arrangements proved overly complex, with execution responsibility shared between the State of Pernambuco, municipalities of Olinda and Recife and the water company, COMPESA. Ability to execute was overly reliant on political support and the varying levels of financial and technical capacity of these local Government agencies. In addition, the Project’s M&E design did not include a clear method for evaluating the degree to which the involved entities’ capacities increased.

Other shortcomings (ICR, p. 6 -7) included: (a) lack of readiness at effectiveness; (b) resulting underestimation of costs of some components; (c) inappropriate disbursement categories (disbursement categories for goods and works had to be added through restructurings); (d) rigidity in the project's cost allocation, which made it hard to adjust design after the price increases, without formal legal amendments; (e) inadequate provision upfront for participatory involvement at the design stage, save for two pilot areas; and (f) lack of provision for commercial space within the area of the housing improvements, leading to some families expanding their homes to accommodate informal business, which was not always well accepted by other residents. Providing for mixed use in the area would have required a change in the urban zoning regulations, which may have been a reason for not incorporating provision for commercial activity.

As an example of lack of readiness: At effectiveness (2003), the Project only had engineering designs for two pilot water service areas (UEs), and the first bidding processes for these works, launched only in 2005, failed because of underestimated costs. The development of the engineering designs for the. additional UEs took two more years to complete (as late as 2007).

Government added other comments on shortcomings: (a) the lack of detailed soil studies in areas where the housing complexes were to be built led to "countless delays" and cost overruns in the execution phase; and (b) the sustainability strategy, which relied on initial subsidy of capital costs of provision of basic infrastructure for low income communities, followed by cost recovery from tariffs and taxes, proved unrealistic since most low-income project beneficiaries were exempt from taxes and tariffs (ICR, p.41).

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
The ICR has the following summary assessment: "The Bank Team demonstrated great flexibility and determination throughout supervision, successfully guiding the Project through years of delays, financial uncertainty, and resettlement booms without losing sight of the development objectives....the Bank team restructured the Project four times to respond to Project challenges and the Borrower’s needs."


However, in the TTL meeting, staff revealed the frequent turnover of TTLs, and the fact that the first TTL had not required progress reporting. Also, in the early stages of the project, disbursements were extremely behind schedule, and the attempt to persuade Government to enforce the cutoff for beneficiary registration was unsuccessful. Project teams in the latter years of implementation appear to have performed better: The ICR pointed out that the Bank Team went beyond pure supervision, advising the Borrower on areas outside of the Project’s immediate scope (such as the proposed public-private partnership for COMPESA); the task team is said to have played a role in facilitating the identification of the parallel European Union urban poverty and social vulnerability reduction initiative in Olinda (The €936,660 “Projeto Comunidade Viva – Projeto de Redução da Pobreza no Município de Olinda ) as well as several other domestically funded initiatives in the project areas. Finally, showing its commitment, the task team is continuing post-closing supervision to ensure all project components are completed satisfactorily, including resettlement of the remaining families without housing. The Borrower’s completion report (ICR, Annex 6) emphasizes the high value that the implementing agencies’ attached to the team’s guidance throughout implementation, drawing on the Bank's global experience with such projects.

Considering the uneven quality of supervision throughout the life of the project, the rating is Moderately Satisfactory.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
Government here refers to the Government of Brazil, who guaranteed the loan, the State of Pernambuco (GSP) and the municipalities of Olinda and Recife.


At the time of preparation, the Project was a clear priority of the Government. In 1999, the GSP and the Federal Government signed the State of Pernambuco Restructuring and Fiscal Adjustment Program (PRAFEP, and this project was one of four external credit operations included. The GSP, demonstrating its strong commitment to the Project oversaw technical studies for institutional and financial arrangements, analyzed project alternatives, and led poverty assessments in the proposed Project areas. The GSP also selected two areas (out of the thirteen) for pilot projects early in preparation, including consultation with the proposed beneficiary communities The GSP was also fully cooperative with the Bank in appraising the Project and made unanticipated concessions--for example, agreement to contract private operators to handle water services in the Project area at a time when discussing possible privatization was very sensitive politically (ICR, p. 6). Toward the project end, when it was clear some activities-- even in the adjusted scope of work-- could not be completed within the funding envelope, the State and municipal Governments increased their counterpart funding by $36m. Obtaining the necessary complementary financing at the federal level led to delays in the full funding required, however, and the implementation of some components is still continuing beyond the project completion date.

The main shortcoming that affected the GSP’s performance was the lack of a formal and efficient enabling environment for metropolitan governance and coordination between the State and the municipalities. (ICR, p. 18-19). Another issue in performance regarded the decision to not enforce the cutoff date for registration of resettlees under the Project. This decision went against the recommendations of the Bank's resettlement policy. The consequence was a 37% increase in beneficiaries requiring resettlement, which swamped capacity and has led to the need to house some in rental housing at Government's expense (ICR, p.19).Since the ICR refers to this as "the Borrower's decision" , IEG places this issue in this section..

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
Institutional responsibilities: The PMU (in State of Pernambuco) was responsible for coordinating the institutional agencies’ actions, monitoring and evaluating implementation, and strengthening the capacity of the municipalities to implement the Project. Implementation of the works, titling, and consultation components fell under the responsibility of the State and the municipalities. The legal agreements provide that the state would oversee and supervise the entire project execution, and be responsible for overall project coordination. Through the Foundation for Municipal Development (FIDEM), the state would execute the first two pilot works of the low-income area investments, after which, the execution of local infrastructure works would be executed and maintained at the municipal level to better ensure sustainability. The agreement signed by the State and municipalities as well as the Loan Agreement signed by the state, and the Project Agreements signed by the municipalities made these arrangements legally binding.

Positive: PMU staff members were able to maintain financial management arrangements compatible with Bank requirements, and help ensure that counterpart funds were available throughout the last years of Project implementation. Project accounts, contracts, and transactions involving the Special Account, were audited each year under terms and conditions satisfactory to the Bank. (As noted in fiduciary section, action on some final audit recommendations remained outstanding at project close, however). Procurement activities were also carried out in accordance with the Bank's Guidelines.
.
Shortcomings in PMU performance included: (i) financial management and reporting shortcomings in the initial years of implementation; (ii) challenges in assuring adequate exchange and flow of information; and (iii) obtaining key documentation from the municipalities in a timely manner. Also, the PMU lacked the ability to harmonize lines of action between the Governor and the Mayors.

Regarding the other implementing agencies, the ICR states that, although the Project’s initial shortcomings were at the State level, later shortcomings were at the municipal level. The Municipality of Olinda lacked capacity and experience in executing large scale infrastructure works and managing resettlement. Recife faced financial shortcomings, slow decision making processes, and difficulties in defining areas for resettlement. The water company, COMPESA, did not give sufficient priority to the Project, possibly because the works were deemed too small in comparison to the company’s portfolio. The Municipalities also struggled to report on the development of the activities in their respective areas. (ICR, p.19).

Nevertheless, in cases where one dimension of Borrower performance is in the satisfactory range and the other is in the unsatisfactory range, overall performance is rated moderately satisfactory because the outcome was in the satisfactory range.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The ICR (p.2, footnote) notes that In the PAD, the Project indicators in the main text and the annex are not aligned. The ICR team chose to present those provided in Annex 1 – Project Design Summary, while correcting an editorial mistake there--"availability of housing” should read “quality of housing.” IEG used the same set of indicators, and notes below the changes that took place since appraisal.


The results framework in the PAD was largely sound, in that major outputs and outcomes were captured: Outcome indicators included increased property values, improvements in quality of housing, and perceptions of quality of life and quality of services (all those indicators were retained and measured at project close). Selected indicators of project efficiency and effectiveness included cost recovery indicators (retained, but as measured, of limited relevance),and two measures not well defined: a program cost to income ratio for beneficiaries (not retained) and a measure of program cost distribution among components (retained, but of limited relevance). Output indicators included a measure of coverage relative to potential coverage (a good indicator, but not retained), works with community members employed (not well-defined), resettlement (indicator of "works completed" a poor proxy) and measures of title registration and physical infrastructure completed (all measurable and relevant).

The ICR also reported on a few additional indicators, one measuring community participation (well-defined) and one measuring decentralization of works execution responsibility (of limited relevance.) The ICR changed a few of the physical infrastructure indicators to make them more relevant (drainage works), dropped those pertaining to dropped project components (retaining walls), and dropped the indicator "Adoption of a streamlined process of land tenure regularization in the informal settlements" giving as a reason that this was under municipal, not State responsibility, In the task team meeting, staff said that this component was essentially dropped from the project, and efforts were targeted on titling of the project beneficiaries, not all residents of the area.

Notably, there were no indicators to measure progress of institutional strengthening of the State or municipal Governments, to carry out and sustain an expanding program of urban development, resettlement, and land titling. An "outcome indicator" that attempted to measure "decentralization structure" of the project only covered assigned implementation responsibility, and not the quality of execution of that responsibility, or progress toward institution-building objectives.

b. M&E Implementation:
The ICR stated (p.8) that M&E implementation was a challenge for the majority of the Project’s implementation, due to low institutional capacity, a challenging institutional arrangement and a lack of familiarity with M&E methodologies. The PMU only began providing the Bank with Project Progress Reports in 2009, after a new project team leader informed the Borrower that it was required in the Loan Agreement. The “SIGMA Management Information System (MIS)” did not meet the needs of the project, being too project management oriented, leading to staff not using it. (TTL meeting). Since 2009, however, monitoring data on social outreach and community participation activities, contracts and works supervision and intermediate and output indicators were provided regularly and in a satisfactory manner, largely as a result of stepped up Bank supervision. There was no mid-term Datametrica study, as the PAD called for, (TTL meeting)

a. M&E Utilization:
According to the ICR (p. 8), the PMU began utilizing M&E to guide their actions only during the last two years of Project implementation. The use of M&E instruments, including progress reports, helped both the Borrower and the Bank proactively address implementation obstacles. The production of the Borrower’s Completion Report, which the PMU developed hand-in-hand with an external consulting firm, demonstrated the PMU’s newfound appetite for M&E, and provided a good foundation for the ICR. The PMU is including M&E as a key instrument to be used in the planned next phase of the Project, the PROMETROPOLE II.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
The safeguards triggered, Environmental Assessment (OP / BP 4.01) and Involuntary Resettlement (OP / BP 4.12), were appropriate given the issues posed by the Project. The Environmental Category was B.
--Environment: During the life of the Project there were no environmental safeguards issues. Rather, positive results in terms of urban environment improvement (not captured through the Project indicators) can be attributed to the Project interventions. The ICR provides several examples (P. 9). The ICR does not mention if the environmental assessment or plan was updated after restructuring.
--Resettlement and titling. The Project was implemented under a Resettlement Policy Framework (RPF) dated November 2001. Additional Resettlement Action Plans (RAPs) were prepared for each specific resettlement project. IEG assumes that this process of "rolling RAPs" meant that the plans were updated after the several restructurings. The ICR states that the Project design underestimated the complexity and scale of the resettlement and titling issues, given the complex federal and municipal legislation in Brazil, and failed to anticipate the need for technical assistance in this area. (ICR, p. 6-7). The resettlement was carried out in stages, but the cutoff dates were not enforced. Despite the advice of the Bank, the implementing agencies, for political and social reasons, opted to accept families for resettlement even when they had arrived just before resettlement began. (ICR, p. 9). The number of families that had to be resettled thus increased by 37% over the life of the project. (A second census had to be conducted.) The ICR does not go into detail on the range of solutions (other than Government's failure to enforce the cutoff date) that might have been considered by Government and WB to meet the high demand for improved housing which placed social pressure on the Government. IEG thus downgrades Government and Bank supervision performance, interpreting the events as non-compliance with Safeguard policy.

Despite the consequences of Government's decision on the cutoff date, the ICR states (p.9) that the agencies carried out the resettlement competently, following best practices: (i) after construction, homes were distributed by lottery among eligible families, and families were allowed to trade locations with each other; (ii) a wide participatory process was held during both planning and implementing stages, integrating community-based groups; and (iii) a team of social workers worked closely with the legal, technical, and engineering teams to present and address the concerns of the community.

Although overall safeguards compliance ratings remained “Satisfactory” throughout the Project’s implementation, cumulative delays and unconcluded housing development and resettlement activities became evident during the last year. At the close of the project, some resettlees were still in rental housing, at substantial cost to the Governments involved (ICR, p 9). (245 rental units in Olinda, with permanent housing planned to be completed by December 2012; 44 houses in Recife, with construction not yet begun). The Bank has committed to continue supervising until the needed housing construction is completed. A positive side-benefit is that the State of Pernambuco and the Olinda municipality have embraced the resettlement approach (described as " a paradigm shift" by officials in Olinda) and are applying it to the Government's PAC scale up program. (ICR, p. 13)

b. Fiduciary Compliance:
The ICR (p.8) states that there were no major fiduciary issues on procurement during the Project’s implementation, and the PMU was mostly in compliance, despite major challenges in procurement posed by the many amendments to contracts required by inadequate planning, problematic engineering designs, delays in implementation, and cost escalations. The ICR stated that the Borrower was well versed in the Brazilian National procurement system, but was not familiar with the Bank guidelines, which led to delays in the civil works’ procedures, due to long internal approval procedures.

In addition to the lack of progress reporting in the early implementation, the ICR (p.8-9) cited the following specific financial issues: (i) delays in the delivery of the 2008 audit report, and delays in the implementation of auditors’ recommendations; (ii) shortcomings in the monitoring and reporting of counterpart funded activities; and (iii) instability in the periodicity and quality of Statement of Expenditures (SOEs) and Interim Financial Reports (IFRs). Financial management (FM) improved after the implementation of several FM action plans agreed between the Bank and the PMU. At time of closing, the rating for procurement and FM was moderately satisfactory. But the last audit identified some issues still being corrected by the Borrower and followed by the project team: (i) aspects of the financial management system (referred to as "SIAFEM", not identified); and (ii) bank conciliations of Recife and Olinda municipalities. Concerns are registered in two management letters sent to the Borrower (February 27 and April 4, 2012), both filed in WBDocs.

c. Unintended Impacts (positive or negative):
A positive side-benefit of the project was that the State of Pernambuco and the Olinda municipality have embraced the best practice resettlement approach developed during the project (described as "paradigm shift" by officials in Olinda) and are applying it to the PAC scale up program. (ICR, p. 13). Also, contractors told supervision teams that 10 to 15 percent of the workers who were hired from the local community acquired marketable construction skills and found careers as construction workers.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Significant
The multiple issues in project implementation illustrated risks of high consequence that were unanticipated. These risks will continue, both as project infrastructure needs to be maintained and as Government scales up with similar investments. 
Bank Performance:
Moderately Unsatisfactory
Moderately Satisfactory
When the rating for one dimension is in the satisfactory range while the rating for the other dimension is in the unsatisfactory range, the rating for overall Bank performance depends upon the outcome rating which in this case was in the satisfactory range. 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
1. Defining project sub-areas for project improvement based on technical aspects, rather than political or administrative boundaries, can improve project performance, by increasing legitimacy, decreasing political interference, and facilitating an integrated public infrastructure improvement approach. (IEG, adding to ICR). In this project, the improvements were based on "sanitation collection units" (UEs).

2. A resettlement plan should be based on objective criteria of need and eligibility, grounded in data collected in a census, usually carried out early in project implementation, if not prior to appraisal. Even if a resettlement plan calls for the implementation to be carried out in stages, the eligibility criteria must be clearly planned and communicated to the Government and the affected population at the outset. The Bank’s Resettlement Policy (OP/BP 4.12) recommends setting a cutoff date at the same time that the census is completed. After this date, no entitlements should be offered to people who settle in the area or make improvements to their homes or businesses. Making these conditions clear would take political pressure off the Government to provide entitlement to new squatters. (IEG, restating lesson in ICR)

3. Planning of Urban Upgrading projects should incorporate a review of the urban zoning regulations, and processes to consult existing and potential residents, to ensure alignment of the regulations with the aspirations of the residents, and the economic development opportunities. (IEG, broadening ICR author's lesson). In this project, the Borrower decided to build single family dwellings and town houses, to preserve the traditional way of living of poor families, while upgrading infrastructure; it neglected to consider the need for formal business spaces. As a result, many new residents expanded their homes to accommodate informal businesses, such as bars, hair dressing and convenience stores, which did not fully satisfy their needs, or the preferences of other residents for mixed development.

4. When assigning implementation responsibility for complex investment programs to inexperienced sub-national agencies, project-specific institutional capacity assessments should be carried out, to determine if components for technical assistance need to go beyond the conventional WB project management processes, to include technical support , including those related to safeguard compliance. (IEG)

5. Preparation for complex urban upgrading projects incorporating resettlement should include review of land use and land titling laws, and plans should be made to regularize tenure of all residents, not just those benefitting from the project, so as to avoid inequities, (IEG)

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

Positive aspects of the ICR were:
1. Largely concise, robust presentation which covers all the subject areas required in the WB Guidelines.
2. Frank presentation of the many implementation issues, the likely causes and the consequences.
2. Highlighting of inconsistencies found in the Project appraisal document (PAD), and the choices made in the ICR presentation and analysis.
3. Good, critical analysis of the PAD's results framework, and adjustment of indicators to better reflect the intentions of the framework, and increase relevance for judging project performance and outcomes.
4. Taking advantage of the availability of Datametrica, the same firm that conducted the PAD economic analysis, to conduct the ICR economic analysis, which likely resulted in efficiencies.

Areas that could have been improved:
--The ICR is not always clear in the definition of the project's boundaries, as it affects the resulting reporting of actual costs, and the judgements on efficacy, efficiency and outcome: (a) it is not clear if the boundaries include the completion of works funded by additional counterpart funding; and b) it is not clear if assessment of outcomes are based on the original definition of components, or revised components. The conclusions drawn from indicators are a mix. A reference on revised components in "Annex 10" is a wrong reference (Annex 10 is the map) and it is difficult to reconcile figures in Annexes 2,8, and 9.
-- Explanation of institutional arrangements were not complete, making the judgements on performance more difficult. The rationale for the ratings of respective performance of the PMU, State, and municipalities were not presented in full.
--Treatment of shortcomings of the Datametrica methodology for the economic analysis were incomplete. The key question of whether the improved economy, general inflation, and construction costs inflation accounted for a significant part of the reported "increase in housing values" was not analyzed. In addition, results of the economic analysis were presented in an unbalanced way--the negative benefit cost ratio for the improvements in the low-income areas was not highlighted, and the presentation of benefits for the resettlees did not include presentation of the cost.
--Explanation of eligibility criteria for families resettling was lacking; indicator refers to "number of families resettled due to geotechnical risk and/or encroachment into environmentally sensitive areas" , but the ICR implies the Government applied looser criteria.
--The currency conversion rate reported as used in the benefit/cost analysis does not match either of the rates in the opening pages of the ICR (rates at approval and at closing)
--Some analytical language was not used precisely, e.g. value per residence was used interchangeably with value per household, and "increase" in housing value was sometimes used instead of the correct term "difference".

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions