Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Nutrient Reduction Project


  
1. Project Data:   
ICR Review Date Posted:
12/20/2013   
Country:
Hungary
PROJ ID:
P074971
Appraisal
Actual
Project Name:
Nutrient Reduction Project
Project Costs(US $M)
 32.0  33.8
L/C Number:
Loan/Credit (US $M)
 12.5  12.3
Sector Board:
Environment
Cofinancing (US $M)
 7.7  7.7
Cofinanciers:
IBRD (redirected unallocated funds from predecessor Budapest Municipal Wastewater project (Loan No.: 4512-HU))
Board Approval Date
  04/18/2006
 
 
Closing Date
12/31/2011 12/31/2011
Sector(s):
Irrigation and drainage (76%), General agriculture fishing and forestry sector (12%), Flood protection (12%)
Theme(s):
Pollution management and environmental health (33% - P) Biodiversity (33% - P) Other environment and natural resources management (17% - S) Water resource management (17% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Richard C. Worden
Judyth L. Twigg Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The development objectives as stated in the Project Appraisal Document (PAD; Results Framework and Section B.2 of Technical Annex 3, no page number provided) was to:

(i) reduce Budapest’s discharge of nutrients (nitrogen and phosphorus) into the Danube River, and consequently into the Black Sea;

(ii) enhance the nutrient trapping capacity of the Gemenc and Beda-Karapancsa wetlands of the lower Hungarian part of the Danube River; and

(iii) serve as a model for similar nutrient reduction initiatives in Hungary and other Danube basin countries.


This description of the project objectives was identical to that given in Schedule 2 of the Global Environment Facility (GEF) Trust Fund Grant Agreement (TF 055978-HU; p. 18). Thus, these three objectives form the basis for IEG’s assessment.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
The Project had three components:

Component A: Construction of tertiary treatment facilities at the North Budapest wastewater treatment plant (WWTP) (Estimated cost: US$ 23.4 million; Actual cost: US$ 26.8 million). A single internationally bid contract was to be issued to decrease the concentration of nitrogen and phosphorous (N&P) in the effluent water. A nitrogen removal technology using activated sludge was to be installed, and reductions in phosphorous levels were to be achieved by adding ferric chloride to precipitate out the phosphorous.

Component B: Wetland restoration works in the Gemenc and Beda-Karapancsa wetlands of the Duna-Dráva National Park (Estimated cost: US$ 6.1 million; Actual cost: US$ 6.5 million). Eleven restoration works were planned in these two areas of global conservation importance, as evidenced by their status as RAMSAR sites (the UN Convention on Wetlands of International Importance). Both are situated entirely within the Duna-Dráva National Park (DDNP) located along the Danube River downstream from Budapest. This component was also intended to establish a comprehensive Monitoring and Evaluation (M&E) system to document the effectiveness and cost-efficiency of project interventions in terms of retaining N&P concentrations within the wetlands and thereby reducing their outflow into the Danube River.

Component C: Dissemination activities to foster replication in Hungary and in the other 10 countries of the Danube River water basin (Estimated cost: US$ 0.4 million; Actual cost: US$ 0.5 million). This component was to finance a comprehensive end-of-project impact evaluation and results analysis of the two interventions (tertiary treatment and wetlands restoration), including a cost-benefit analysis. The results of these studies were to be used as the basis for dissemination, replication, and knowledge-sharing activities at a regional workshop, public communication campaigns, and on the project’s website and the GEF-funded International Waters (IW) Learn Initiative.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost: The total estimated cost for the project was US$ 32.0 million; the actual cost was six percent (6%) above this at US$ 33.8 million. Of this total amount, cost overruns occurred across all three project components, but due to its absolute dollar size compared to the other two components, the cost-overs for Component A to finish the tertiary treatment facilities at the North Budapest WWTP of US$ 3.4 million accounted for most of this. The other two components only contributed US$ 0.51 million to this total. Roughly half of the cost overruns were covered by “project contingencies” of US$ 2.09 million, with the other half covered by US$ 2.01 of additional local financial support.

Financing: The GEF provided US$ 12.32 million out of its estimated contribution of US$ 12.50 million. A Project Agreement between the Bank and the Municipality of Budapest authorized the use of US$ 7.7 million in loan savings and unallocated funds from the Loan Agreement (IBRD-45110 and -45120 loans) of the predecessor Budapest Municipal Wastewater Project (P-008497) to complete the tertiary treatment facilities at the North Budapest WWTP under Component A of this project.

Borrower Contribution: The Government provided US$ 13.8 million to the project, 17% above its estimated contribution of US$ 11.8 million (of which US$ 10.4 million was to come from the Municipality of Budapest and US$ 1.4 million from the Government).

Dates: The project was appraised in October 2005 but did not become effective until August 11, 2006. Some revisions were made by the Bank and Borrower to the Intermediate Outcome Indicators in June 2008, but these were not formally revised later during the project’s restructuring on March 25, 2011. The changes made formally in 2011 did not affect the project’s objectives, components, or sole Global Outcome Indicator (Overall reduction of the nutrient flow into the Danube River and the Black Sea). They only reorganized Hungary’s implementing ministries and added two Intermediate Outcome Indicators to the Results Framework. These changes were made prior to the issuance of the Bank’s new Investment Lending guidelines (released in October 2009), which required formal restructuring for such changes, which may explain why they were not formally revised. The Mid-Term Review was conducted nearly two years later than planned in November 2009, but due to a 10-month delay between the Bank’s appraisal and approval of the project and its effectiveness date in August 2007, the planned date for the Mid-Term Review was only 16 months into project implementation. The project closed as scheduled on December 31, 2011.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High: The 2002 Country Assistance Strategy for Hungary specifically identified nutrient reduction as an objective for the Bank’s investment and assistance (CAS, p. 18). This project built on a Specific Investment Loan for the Budapest Municipal Wastewater Project (US$ 31.6 million;1999-2008) that the Bank had nearly completed to assist Hungary in reducing its nutrient loadings in the Danube River, to upgrade Hungary’s wastewater utilities, and to strengthen its compliance with European Union environmental standards. Managing the amount of nutrients, principally nitrogen and phosphorous (N & P), that enter the Danube River is one of the most significant interventions that can be undertaken to improve the regional environmental quality of the surrounding watershed. Ten countries border and drain into the Danube River basin, including the Republic of Hungary, which represents about 11% of the river’s drainage area.

Of particular interest to both the Bank and the Government was a second project objective to experimentally assess the role of floodplains and wetlands to serve as a cost-effective alternative to more expensive and technologically intensive conventional wastewater treatment plant (WWTP) systems. This was seen as an opportunity to quantify and compare the costs and benefits of actively engineered WWTPs with more passively engineered (i.e. sluices and weirs) natural systems. A third objective was added to ensure that the resulting information would be shared with other countries in the Region so that they might consider implementing similar environmental management interventions in the future.

Since Hungary “graduated” from the Bank’s system of financial support to developing countries during the spring of 2007 (while maintaining an active partnership with the Bank on technical assistance and analytical services), it has not updated its 2002 CAS with the Bank. Therefore it is not possible to assess the relevance of the project’s objectives at the time of the project’s closing at the end of 2011. However, the ICR notes in the “Implementation Stage” (Section 2.2; no page number) that "the public, and especially the townships and communities closest to the proposed works, were generally supportive of the project’s objectives and recognized the values to the environmental quality of the Danube River and the Black Sea, and to fisheries and wildlife.” This would seem to indicate that the project’s objectives are still relevant and continue to contribute significantly to the protection of this regionally important natural resource.

b. Relevance of Design:
Substantial. The three components contained in the project’s design were directly relevant to achieving its objectives, and there was a solid causal link between inputs, activities, outputs, intermediate outcome indicators, and the Global Outcome Indicator and the three project objectives. Each of the three project components (and their associated indicators) reflected distinct and important aspects of achieving project objectives. All but one had measurable associated targets. The activities were logically linked to achieving the desired outcomes, and resources were realistically estimated (although slightly under-estimated), and adequately supported each one of the components.


4. Achievement of Objectives (Efficacy) :

PDO #1: Reduce Budapest’s discharge of nutrients (nitrogen and phosphorus) into the Danube River, and consequently into the Black Sea: substantially achieved.


Outputs:

  • Construction of tertiary treatment facilities at the North Budapest Wastewater Treatment plant (WWTP) to remove Nitrogen and Phosphorous from effluents released into the Danube River.

Outcomes:
  • 93% achievement (3,720 tons/year reduction of total nutrient loadings released to Danube River) of Global Environmental Outcome Indicator target of “around 4,000 tons/year." The ICR states that measurements were taken while the WWTP was operating below capacity. Therefore, it is expected that when the WWTP is operating at full capacity the target will be met and/or exceeded.
  • The first Intermediate Outcome Indicator associated with this objective was: “Annual reduction of nutrient discharges from the North Budapest WWTP of about 70% of Nitrogen (N) and about 50% of Phosphorus (P) by the end of the Project.” The targets for removing N & P from the plant were exceeded by more than 25% in both cases, as well as for Biological Oxygen Demand (BOD), which is considered a more dynamic biological indicator of on-going water quality conditions than more static chemical indicators.
  • The second Intermediate Outcome Indicator was: “Average operational cost of the nutrient reduction process in the North Budapest WWTP.” No target value was provided in the PAD. However, the ICR states that the original target values (from appraisal documents) were US$ 0.03-0.05 per cubic meter of treated effluent. The actual values obtained were 43 – 59 times higher at US$ 1.78 - 2.15 per cubic meter of treated effluent. Thus, the target for this indicator was not met, although this is partly attributed to imprecise cost estimates conducted during preparation and appraisal of the project.

PDO #2: Enhance the nutrient trapping capacity of the Gemenc and Beda-Karapancsa wetlands of the lower Hungarian part of the Danube River: modestly achieved.

Due to the inherent difficulty and cost to accurately measure the nutrient retention of wetlands, a less direct but more measurable proxy (the number of hectares of wetlands rehabilitated by the project) was used to measure achievement of this project objective.

Outputs:
  • 62 “passive” civil engineering works were completed in five sections of the Gemenc and Beda-Karapancsa wetlands covering 4,300 square hectares (against a target of 10,000 square hectares).
  • A “fully operational” comprehensive monitoring and evaluation (M&E) system was accomplished (ICR, discussion of Component B Outputs, no page number given).


Outcomes:
  • Retention of partially treated wastewaters flowing through the Duna-Dráva National Park wetlands was increased from only 28% of the target value of 43,000 cubic meters per hectare per year to over 231% of the target value (ICR, Section F: Results Framework Analysis, no page number).
  • A fully operational M&E system was developed by the project, but the only information given regarding its utilization during the project or the role it played in informing project management was with regard to the impact evaluation study that was prepared at the end of the project.

PDO #3: Serve as a model for similar nutrient reduction initiatives in Hungary and other Danube basin countries: modestly achieved.

Outputs:
  • A Cost-Benefit Analysis and Impact Evaluation Study of nutrient reduction by means of wetland restoration compared to WWTP tertiary treatment were carried out. These were discussed at an end-of-project workshop involving stakeholders from Hungary as well as from the broader Danube region, and were disseminated electronically through the International Waters (IW) Learn facility.

Outcomes:
  • The Cost-Benefit Analysis showed that the actual cost of nutrient removal in the DDNP wetlands (US $2,623 per unit of nutrient-laden water retained) was nearly 11 times higher than the estimate of US $240 per unit retained. This was even higher than the cost of tertiary treatment at the North Budapest WWTP (US$ 2,230 actual), which was more than double the estimated amount per unit of nutrients treated (US$ 1,060 estimated).
  • The Impact Evaluation Study provided an improved understanding of the role of nutrient reduction technology and the role of wetlands in performing similar services for this region than had been previously known. The Study identified the technical and environmental difficulties encountered in conducting and interpreting consistent, repeated measurements of nutrient dynamics along the Danube River.
  • Although both of these indicator targets were accomplished, they represent outputs rather than outcomes reflecting the fact that while they may have been disseminated at the final project conference, through study tours, and on the project’s website, it cannot be reasonably concluded that they therefore “served as a model for similar nutrient reduction initiatives in Hungary and other Danube basin countries." There is not evidence that the project's activities were replicated in those places, nor is there evidence that the project's experience and analyses based on that experience were meaningfully transmitted to serve as inputs into decision-making elsewhere. First of all, such studies are locality-specific and must be conducted for every intervention in sensitive areas. Second, there was no persuasive information provided substantiating the claim that the project catalyzed these other EU initiatives, which otherwise would not have occurred without this project (the counter-factual). Third, due to delays encountered in obtaining the necessary licenses and permits from various Hungarian agencies with jurisdiction in the DDNP, the project was not able to begin civil works in the wetlands until March 2011. Fourth, the implementing agencies have not been able to obtain the necessary funds to continue the biological monitoring program (for N & P retention rates) since the project’s closing. This has constrained the project’s ability to build the evidentiary case for adopting (or not adopting) similar interventions elsewhere.

5. Efficiency:

Efficiency is rated modest.

Economic Rate of Return and Estimate of Environmental Benefits: At appraisal, the Economic Rate of Return (ERR) for the North Budapest WWTP (Component A) was estimated at 22% while that for Component B (the Duna-Dráva National Park wetlands) was estimated at 72%. The ERRs actually achieved for Components A and B, based on the economic analysis at the end of the project, were 12.7% and 13.21% respectively. These were much lower than expected, due in large part to wildly inaccurate cost estimates during preparation and appraisal. For example, the cost of expanding the North Budapest WWTP was under-estimated by 70% or US $13,339,407. Overall, however, the estimated environmental benefits exceeded the costs of abatement, estimated at 1.31 for Component A and 1.86 for Component B.

Cost-Effectiveness Assessment: In terms of cost effectiveness, the unit cost of abatement was estimated at US$ 1,060/ton of treated effluent for the North Budapest WWTP component and US$ 240/ton for the Duna-Dráva National Park component. However, in both cases, the unit costs of nutrient removal calculated in the final economic assessment were much higher than those at appraisal (as described above in the previous section: US$ 2,430 and US$ 2,623 per ton for Components A and B, respectively). The lower than envisaged efficiency was linked to the higher costs of construction and operation than estimated, especially for the Duna-Dráva National Park wetland component. These gross under-estimates had serious repercussions for the project’s operational efficiency as well. The cost overruns had to be picked up by the Government, which agreed to do so for Component A, but did not agree to do so for the Duna-Dráva National Park wetlands restoration component (possibly due to its higher than expected costs, some resistance by a local hunting group, and as yet undocumented benefits arising from its implementation). This resulted in the downsizing of the originally proposed civil works in the Duna-Dráva National Park wetlands from 11 areas down to five, which necessitated a long delay as a detailed and complex “Priority Matrix” scheme was developed, discussed, and final selections were made by project implementers. This demanded the inefficient use of time and staff resources that would not have been necessary had more accurate cost estimates and better institutional analysis been done of multiple Government agencies with overlapping jurisdictions for granting the required licenses during preparation and appraisal. These factors resulted in higher costs and long delays in the start of civil works and monitoring programs for Component B until the last year of project implementation, too short a time to obtain the desired level of information needed to convincingly document the economic and environmental benefits of this passive engineering approach of using natural wetlands to treat municipal waste waters over more traditional advanced WWTP technologies.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
22%
98.7%
ICR estimate:
Yes
12.7%
98.5%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The Relevance of Objectives is assessed as highly relevant to the 2002 Country Assistance Strategy for Hungary, which specifically identified nutrient reduction as an objective for the Bank’s investment and assistance. Relevance of Design is rated as substantial, as all three components contained in the project’s design were directly and logically linked to achieving the project’s objectives and there was a solid causal link between inputs, activities, outputs, intermediate outcome indicators, and the Global Outcome Indicator with all three project objectives. Efficacy of achieving the first project development objective is rated as substantially achieved because the targets established in the Results Framework were either nearly met or exceeded for nutrient loadings released into the Danube River as a result of civil works conducted with project funds. The second and third project objectives are considered only modestly achieved because the target for enhancing the nutrient trapping capacity of 10,000 hectares of the Gemenc and Beda-Karapancsa wetlands was only achieved for 4,300 hectares, and the case for attribution was not made with regard to the Cost-Benefit Analysis and Impact Evaluation Study serving as models for decision making for similar nutrient reduction initiatives in Hungary and other Danube basin countries. Efficiency is rated as having been modest due to lower than expected financial cost-effectiveness rates and operational inefficiencies.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

Maintaining the operation of advanced waste water treatment facilities at the North Budapest WWTP is a requirement for the Government of Hungary in order for it to comply with EU wastewater treatment standards and regulations. Thus, the likelihood that it would not meet these water quality standards is low. Improving the water quality of the Danube also enjoys broad popular support among the people of Hungary, according to the ICR (Section 2.2; no page number), further decreasing the political risks of retreat from this level of wastewater treatment. The North Budapest WWTP is also generating 90% of its energy requirements from biogas generated by the anaerobic digestors, which is a significant cost savings. Low operating costs will keep water utility rates and bills low, thereby lessening resistance from the public on financial grounds. The technological skills and institutional support required to run tertiary water treatment facilities are well established in Hungary. Thus, there are no significant risks posed to the continued operation of this facility.

However, the risk to development outcome associated with effectively managing the rehabilitated wetlands in the Duna-Dráva National Park effectively to prevent eutrophic conditions from developing is considered moderate given the current limited technical capacity of the Duna-Dráva National Park Directorate personnel and the high costs of sustaining a sophisticated water quality management and monitoring program in the medium- to long-term (ICR, Section 4: Assessment of Risk to Development Outcome, no page number). This also suggests that the required biological monitoring information will not be available to adequately assess the dynamics and uptake of N and P, and thus document the effectiveness of wetland performance to reduce nutrient loads over time. The DDNP Directorate has acknowledged its lack of technical capacity and resources to perform the level of technical monitoring needed to adequately maintain long term data on wetland nutrient dynamics.

Finally, the potential for this work to serve as a model for replication elsewhere remains low given the higher than expected costs and lack of monitoring data on nutrient retention levels that have been achieved. The case has simply not yet been made in a convincing manner for application in other similar situations in the Black Sea region.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:
The Bank assisted in the preparation of the necessary assessments and studies, including the Environmental Impact Assessment and Environmental Management Plans. The Bank accurately assessed the strengths and weaknesses of Government implementing agencies, providing additional support to the inexperienced Project Implementation Unit for Component B (the Directorate responsible for the Duna-Dráva National Park wetlands), and established an umbrella Project Management Unit in the more experienced Municipality of Budapest and its water authority (the BMSC). The roles and responsibilities of each of the project partners were well defined during preparation and were spelled out in the PAD and Grant Agreement. Implementation structures were clear regarding reporting requirements to the Bank for each of the project partners, with clear and transparent lines of authority and funding for each of the components.

The project design was also novel and forward-looking in terms of seeking monitoring systems and cost-benefit analysis to test important environmental and economic comparisons between actively engineered wastewater treatment systems (i.e., the North Budapest WWTP) against the role that natural systems, such as wetlands, can play in the natural processing of wastewaters. At the same time the project was well-designed in terms of reducing transaction costs by working with a single national government while testing and comparing innovative approaches with regional impacts and benefits, costs that otherwise would have likely been much higher had multiple countries been involved.

However, the contracted bids for the upgrade of the North Budapest WWTP under Component A came in about 70% more than the Project had originally anticipated (Borrower’s ICR, “Original Components” under Section 1, no page number). Being by far the single largest outlay of resources for works under the project, this was a serious underestimation of costs. The resulting shortfall in project funds was covered midway through the project by the Government. In addition, the costs and bureaucratic hurdles of constructing the civil works to rehabilitate the Duna-Dráva National Park wetlands were also grossly under-estimated. The project would have benefited from engaging a riparian, aquatic, or wetland ecologist with expertise on the fate and transport of nutrients in the wetlands environment to help guide early planning and design for long term environmental monitoring associated with Component B. Works on-the-ground did not begin until the last year of the project in the spring and summer of 2011 (Borrower’s ICR, “Key Factors Affecting Implementation and Outcomes” under Section 2, no page number) due to long delays (averaging well over one year) in obtaining the Owner’s Permission, Environmental License, Water License, Forestry Authority License, and Construction Permit. The fact that several institutions and authorities had jurisdiction over project implementation should have been well known to the Bank and Borrower at entry. While the final beneficiary of the wetlands work was the Duna-Dráva National Park Directorate, the forested areas were managed by the GFG Co. whose consent was essential for project implementation. However, the GFG Co. repeatedly voiced its opposition from project design throughout implementation, and this could have and should have been avoided through better communication during preparation or early in the implementation phase. Finally, the main shortcoming at entry was related to the design of the M&E Plan and the Results Framework, which did not capture all aspects of the project’s objectives, leaving out important outcome measures regarding the potential of wetlands to retain nutrient loadings, and to sustain follow-up monitoring to document its effectiveness and promote its replication elsewhere in the Black Sea region.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
World Bank supervision missions were regularly conducted. Supervision and monitoring reports, including ISRs, supported the project’s staff to adjust to both anticipated and unanticipated challenges. The Bank’s Sector Manager and Country Director demonstrated strong managerial awareness and effectiveness in making recommendations to the Task Team Leaders over the course of project implementation, and ratings were adequately flagged by the TTLs in the ISRs and Aides Memoire. The project's Mid-Term Review identified critical issues needed to complete agreed-upon activities within the designated time frame, in particular, the pace of completing the wetland rehabilitation works in the Duna-Dráva National Park.

On the other hand, the Bank did not formally restructure the project to reflect the additions/changes made to the project’s Outcome Indicator and Intermediate Outcome Indicator targets. It also may have been wise to extend the project given the extraordinary delays in starting the civil works to rehabilitate the ecological functioning of the Duna-Dráva National Park wetlands and beginning the biological monitoring protocols in the last months of the project, given their importance in making the case for replicating similar low-cost, low-tech approaches elsewhere in the Black Sea region. Secondly, local resistance to those works by a local hunting association (the Gemenc Forestry and Gaming Company) and by Government agencies with overlapping jurisdiction in the Duna-Dráva National Park, such as the GFG Company, slowed down implementation to the point of compromising the project's ability to achieve its objectives. By the ICR’s own admission, this opposition and the misinformation campaign that it engaged in should have been addressed earlier and more assertively during implementation of the project. Finally, the frequent turnover of project TTLs (there were six TTLs during this five-year project) was a concern among the Borrower and a number of the partners that contributed to a feeling of discontinuity and had an impact on counterpart project staff morale.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The ICR states that the Government “committed adequate staff time and resources in preparing the project” (Section 2.1: Project Preparation, Design and Quality at Entry, no page number), and that the Ministry of Finance (which later became the Ministry of National Economy) and the Ministry of Rural Development were “generally committed to the project and managed the financial aspects of the project to a satisfactory degree”…”and that reporting and transparency were all well managed and documented.” (Section 5.2(a): Government Implementation Performance, no page number.)

The Government paid for a significant portion of the cost overruns for the North Budapest WWTP, which were more than US $13.3 million above the original estimated cost of US $17.7 million. The Government’s contribution for the plant alone was not specified in the ICR, but the Government’s total contribution to the project exceeded the original estimated amount by US $2.01 million or 17%. However, even though the net present value of total capital and operations & maintenance costs for the WWTP was nearly twelve times the net present value for the wetlands rehabilitation works conducted under Component B, the Government chose not to raise additional counterpart funds for those rehabilitation works. Moreover, delays in permitting and licensing to conduct the wetland works had real outcome impacts in terms of the project not being able to document the effectiveness of those passive engineering interventions. The Government also did not allocate budget funds to continue biological monitoring efforts begun in the Duna-Dráva National Park wetlands once project funds ran out, which point to limited commitment on the Government’s part to this alternative “soft engineering” approach to wastewater treatment.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
A Project Agreement (under TF055978-HU; May 15, 2006) was signed between the Bank and the Municipality of Budapest (with the Budapest Municipal Sewerage Company acting as its implementing partner agency) to maintain the Project Implementation Unit for Component A (upgrading the North Budapest wastewater treatment plant). The Duna-Dráva National Park Directorate and the South Trans-Danubian Environmental Protection and Water Management Directorate were the implementing agencies for Component B of the project, while the Project Management Unit had responsibility for the information dissemination and outreach efforts.

Technical and financial aspects of project implementation were satisfactorily reported by the Project Management Unit and project component-specific Project Implementation Units in quarterly and annual reports to the Bank. The Municipality of Budapest and its Water Authority (BMSC) established enhanced chemical and biological monitoring protocols, which were effective in measuring N & P sequestration. The Municipality of Budapest had adequate experience from previous projects with the World Bank and was especially well positioned to support the implementation of Component A. However, while the Duna-Dráva National Park Directorate continued some aspects of ecological monitoring beyond chemical monitoring (especially for wildlife), it does not have the technical capacity to continue biological monitoring of N and P dynamics in the Duna-Dráva National Park wetlands. No explanation was given as to why the Duna-Dráva National Park Directorate has not been able to gain the requisite technical capabilities or secure adequate funding to continue such monitoring efforts, which are critical to accurately and convincingly document their nutrient retention potential and cost for possible adoption elsewhere in the region.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The Ministry of Rural Development designed and was responsible for the overall monitoring and evaluation (M&E) process for project outputs and outcomes, with information provided by each of the project component implementing partners. The main shortcoming of the M&E design was the failure to identify sufficient outcome and intermediate outcome indicators to measure achievement of the second project objective apart from the first in terms of nutrient retention, as well as measuring the third project objective in terms of outcomes rather than outputs. The one outcome indicator at approval only captured the first two objectives collectively, but did not specify the annual water flow or increase in the quantity of nutrients (N & P) retained by the Duna-Dráva National Park wetlands (40% target for each by the end of the project). Some of the targets should have been better defined, and one had no target at all associated with it. The ICR fairly notes that reliable data on baselines and consequently on targets during preparation and appraisal may have been difficult to determine. Hence, it was found necessary during implementation to refine some of the indicators or targets based on actual conditions on the ground. However, it should also be noted that these revisions should have been formalized by the project team.

b. M&E Implementation:
The Municipality of Budapest and its Water Authority (BMSC) established highly effective enhanced chemical and biological monitoring protocols for measuring the effectiveness of tertiary treatment in removing nutrient loadings from its effluents. However, the nutrient retention monitoring protocols developed under Component B were so sophisticated that the implementing partners lacked the requisite technical expertise and equipment to measure them. This necessitated contracting out highly specialized experts from the Budapest University of Technology and Economics to implement them toward the end of the project, and delayed the collection of such monitoring data until only a few months before the project closed.

a. M&E Utilization:
The M&E system established for the first project component for the North Budapest WWTP was well utilized by plant managers to gauge the effectiveness of the new advanced treatment facilities at the plant and provided very accurate and useful cost and nutrient retention data to project implementers and Bank staff. The M&E system for Component B was far less effective due to the delays experienced in getting it operational. Given the novelty of the passive engineering approach that was taken to restore degraded wetlands to their previous functioning, and the importance of that information in making a paradigm shift in engineering thought possible throughout the region, this represented a significant missed opportunity to inform other water resources managers of its costs and benefits. There is now little reason to hope that it will yet be rectified due to funding and technical constraints to continue cost-benefit analysis and nutrient retention monitoring and modeling activities.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
There are no issues of non-compliance mentioned in the ICR, but there is a slight discrepancy between the PAD and the ICR about the number of Safeguard policies triggered by the project. The ICR states that four policies had been triggered, the same three as the PAD (Environmental Assessment (OP 4.01), Natural Habitats (OP 4.04), and Projects on International Waterways (OP 7.50)) plus the Safety of Dams (OP/BP 4.37) policy, which was not triggered in the PAD. Since there were no dams more than 15 meters in height built during the project and no upstream dam infrastructure failure could impact the planned investments, it is not clear why the ICR states that this Safeguard policy was triggered “during the life of the project” (Section 2.4 Safeguard and Fiduciary Compliance, no page number). The only structures engineered and built by the project were sluices and weirs to maximize the retention of influent waters laden with nutrients into the Duna-Dráva National Park wetlands while still allowing them to accommodate excessive flows. The environmental impact assessments, environmental management plans, and other studies carried out during preparation indicated that no significant negative impacts on the environment were expected as a result of project implementation, and minor, temporary impacts were addressed with mitigation measures spelled out in the environmental management plans. The wetland rehabilitation civil works were expected to provide greater benefit than risk in terms of the ecological functioning of those areas and increase the secondary positive effects of increased biodiversity and productivity.

Specific provisions were included in the environmental management plans to address Natural Habitat issues to ensure that the wetland rehabilitation works were properly executed to avoid disturbing migratory species or reduce biodiversity. Official notification under the International Waterways policy (OP 7.50) was waived via a memorandum from the Bank’s Europe and Central Asia Regional Vice President on March 18, 2005. Nonetheless, the other 10 neighboring countries located within the Danube River watershed, which are signatory members of the Danube Convention, were informed of the project within the scope of the Danube Commission through the priority list of the Joint Action Program.

b. Fiduciary Compliance:
Financial Management: The ICR notes that “Fiscal oversight of the project was rated as satisfactory” (Section 2.4: Fiduciary Issues: Financial Management, no page number). As evidence of this, the ICR cites “the consistent flow of funding to the implementing agencies” (ibid). However, project disbursements were slow during the first three years of the project (only US $1.02 million), putting the project at risk. Subsequently, disbursements continued to be uneven throughout the project, with long periods of time of very low disbursement (see Table G: Ratings of Project Performance in ISRs of Section F: Results Framework Analysis, no page number) punctuated by one very large disbursement of US $5.8 million at the end of 2009. Quarterly Financial Management reports by the Ministry of Rural Development to the Bank were well documented, clear and transparent, and the Bank received acceptable audit reports for each of the project years from 2006 through 2011.

Procurement: Procurement for the project was rated by the ICR as satisfactory because the Project Management Unit and its partner agencies for each of the three project components systematically and accurately reported procurement of all goods and works in contracts and consultant TORs, baseline studies and licensing reports, and modified relevant parts of the procurement plan. The "turn-key" contract for the North Budapest WWTP plant was successfully procured using the International Competitive Bidding method, in compliance with Bank standards. Quarterly Project Management Reports prepared by the Project Implementing Units and partners were comprehensive and accurate in reflecting procurement aspects.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Unsatisfactory
Modest achievement of the second and third PDOs coupled with modest efficiency. See Section 6. 
Risk to Development Outcome:
Negligible to Low
Moderate
Risk of properly managing rehabilitated wetlands under Component B remains moderate, and risk of not replicating approach elsewhere is high. See Section 7. 
Bank Performance:
Moderately Satisfactory
Moderately Unsatisfactory
Poor cost estimates for North Budapest WWTP and for wetlands civil works coupled with lack of adequate coordination/communication with Government agencies with overlapping jurisdiction seriously undermined project’s ability to achieve its PDOs. See Section 8. 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
Given their sensitive nature, project interventions in wetlands and other special natural areas require additional time and effort. Due to their drastically reduced coverage throughout the world (> 90% reduction in the last 100 years alone) caused mainly by their physical conversion to agriculture and urban sprawl, the world’s remaining wetlands are often protected by special laws and public concern. They therefore present a difficult regulatory environment in which to work, and the Bank has developed a specific Operational Policy (OP 4.04 for Natural Habitats) to provide additional safeguard protections for such areas. IEG believes that additional time, resources, and effort should be programmed into project preparation and implementation to accommodate anticipated and unanticipated contingencies, such as acquiring work permits or licenses and addressing any local opposition. Although the PAD was clearly cognizant of these hurdles and the potential delays in obtaining the required licenses to conduct the civil works under Component B that might arise, they grossly underestimated the time and effort it would take to obtain them. It may have been a good idea in retrospect to have looked at average time frames for processing permits ahead of time during preparation. The lesson learned here is to estimate preparation and implementation lead times for such works in sensitive areas very conservatively and to scale back targets for project objectives accordingly. In addition, extensive consultations with Governmental and NGO entities with jurisdiction or strong interests at stake in a sensitive area should ideally be conducted ahead of time to identify and resolve any outstanding issues prior to beginning project interventions.

Additional time, resources, and expertise should be programmed into projects when implementing novel approaches. Related to the first lesson, any time that a new and innovative approach is proposed on a project, especially one that challenges long-accepted methods or requires a shift in mentality or mindset, it is imperative that additional time and expertise among respected members or groups within that same discipline be incorporated into the design and implementation of the project if the paradigm shift is to be successful. In this case, it took the professional qualifications and legitimacy of an “Expert Panel” of scientists and engineers to persuade several Government agencies, land owners, and local advocacy groups opposed to the project’s works in the Duna-Dráva National Park wetlands to accept the project and drop their opposition to it.

Improved Bank preparation and appraisal, as well as consistent follow-up throughout implementation, are crucial. A number of Indicator targets were set during preparation and appraisal in an apparently arbitrary manner without having been sufficiently grounded in the prevailing political setting, institutional capabilities and bureaucratic rivalries, climatic variation (especially rainfall), or realistic time frames. The ICR notes that later during project implementation, the high rate of turnover among Task Team Leaders (six TTLs during a five-year project) had a demoralizing effect on counterpart staff and was “intensely frustrating” to them, according to interviews with Bank staff and Borrower comments in their summarized ICR. This high turnover by Bank TTLs (compared to other bilateral donors) was seen as a logical response to the existing set of incentives within the Bank’s own performance evaluation system for operational staff, which rewards getting projects approved and funds disbursed over good performance and results achieved. It also undermines accountability and ownership of project management when there is high staff turnover, and may have contributed to the Bank’s failure to properly seek the required approval by senior Bank management and complete the restructuring procedures. This not only hampered project efficiency, but also contributed to the project’s inability to reach two of its three objectives.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is well organized and clearly written, permitting a comprehensive review of the project to be conducted by IEG. There are a few minor discrepancies in the presentation of project costs by the lender and Borrower, in the Intermediate Outcome Indicators of the Results Framework from the PAD as opposed to those reported in the ICR, and in the number of Safeguard policies triggered by the project. The objectivity of some of the ratings presented is questionable, and some key information from the Summarized Borrower’s ICR at the end of the ICR is not discussed in the main text of the report. Overall, however, the Report is thorough, objective, and illuminating.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions