Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Integrated Management Of Aquatic Resources In The Amazon (aquabio)


  
1. Project Data:   
ICR Review Date Posted:
12/09/2013   
Country:
Brazil
PROJ ID:
P066535
Appraisal
Actual
Project Name:
Integrated Management Of Aquatic Resources In The Amazon (aquabio)
Project Costs(US $M)
 17.13  Missing from NCO
L/C Number:
Loan/Credit (US $M)
 7.18  0.11
Sector Board:
Environment
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  06/13/2006
 
 
Closing Date
02/28/2012 03/26/2012
Sector(s):
General agriculture fishing and forestry sector (60%), Sub-national government administration (20%), Central government administration (20%)
Theme(s):
Land administration and management (25% - P) Biodiversity (25% - P) Water resource management (24% - P) Environmental policies and institutions (13% - S) Participation and civic engagement (13% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
April Connelly
Kristin Hallberg Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
NOTE: THIS REVIEW PERTAINS TO A NOTE ON CANCELLED OPERATION.

PDO
According to the PAD (pg. 9) the project development objective was "to support the main streaming of a multi stakeholder, integrated management approach to the conservation and sustainable use of freshwater biodiversity in public policies and programs in the Brazilian Amazon River Basin. This would be achieved in part through generation and dissemination of subregional experiences that promote and facilitate the adoption of an integrated management approach in the whole Amazon Basin."

According to the grant agreement (pg. 23) " The objective of the Project is to support the main streaming of a multi stakeholder, integrated management approach to the conservation and sustainable use of freshwater biodiversity in the Recipient’s Amazon river basin, ultimately contributing to: (a) reducing threats to the integrity of freshwater ecosystems in the Recipient’s Amazon river basin; (b) assuring the conservation and sustainable use of freshwater biodiversity; and (c) promoting the expansion and replication of integrated management of aquatic
resources in the Recipient’s Amazon river basin over the long term."

GEO
The global environmental objective was "to reduce threats to the integrity of freshwater ecosystems in the Brazilian Amazon and assure the conservation and sustainable use of its freshwater biodiversity of global importance." (PAD, pg. 9)

The project was financed by a grant from the Global Environment Facility (GEF) that was implemented by the World Bank but it did not receive World Bank financing. In accordance with the guidelines for assessing stand-alone GEF projects, this review uses the Global Environmental Objective as opposed to the Project Development Objective as the basis of assessment.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
Component 1: Planning and Public Policy (Appraisal Estimate: US$ 1.26 million; Actual costs not available in the ICR). This component aimed to develop and implement Action Programs for the integrated management of aquatic resources in three sub-basins of the Brazilian Amazon, generate replicable experiences that could become permanent public policies, with positive impacts on aquatic biodiversity, on the reduction of conflicts among various users of natural resources, and on the improvement of local communities’ living and working conditions. This component supported: (a) carrying-out of detailed diagnostics of each of the three project target areas, and the elaboration and implementation of sub-basin Action Programs; (b) the development and implementation of institutional mechanisms for integrated management of aquatic resources in sub-basins and; (c) the development of a financial strategy and mechanisms to provide financial resources for the full implementation of the Action Programs in the long-term.

Component 2: Demonstration Activities (Appraisal Estimate: US$ 6.43 million; Actual costs not available in the ICR). This component aimed to generate experiences and lessons learned, including new technologies or production systems, on how to incorporate freshwater biodiversity concerns into various productive activities, providing inputs for the development of Action Programs for integrated management of aquatic resources. The component supported: (a) demonstration sub-projects that mainstream freshwater biodiversity in productive activities and; (b) other activities, financed under the re-directed baseline, that create an enabling environment for the main streaming of freshwater biodiversity in productive activities.

Component 3: Building Capacity (Appraisal Estimate: US$ 3.67 million; Actual costs not available in the ICR). This component aimed to prepare stakeholders, in particular local individuals and institutions, to be able to actively participate in the formulation, implementation, and monitor strategies and action programs aimed at the conservation and sustainable use of freshwater biodiversity and water resources in the project areas. The component supported: (a) environmental education, (b) training of rural extensionists and local people on sustainable methodologies and technologies, (c) institutional and individual training for the formation of partnerships and conflict resolution and; (d) support for the establishment of decision making mechanisms and conflicts resolution over the use of aquatic resources in the project target areas.

Component 4: Project Management, Monitoring and Evaluation and Information Dissemination (Appraisal Estimate: US$ 5.80 million, Actual costs not available in the ICR). This component supported activities to coordinate, manage, and monitor actions developed under the scope of the project, foster integration among the various components and with other related projects and programs, indicate possible needs for changes in project implementation, and disseminate results at local, state, national and international levels. It also supported the implementation of a project physical-financial monitoring system and the development and implementation of an Information System on Aquatic Biodiversity.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Total project cost was estimated at US$ 17.13 million, to be financed by a Global Environment Facility grant of US$ 7.18 million and recipient co-financing in the amount of US$ 9.95 million, of which: (i) US$ 6.78 million was from the Government of Brazil; (ii) US$ 2.02 million was from re-directed baseline; (iii) US$ 0.48 million was
from the Government of Mato Grosso; (iv) US$ 0.59 million was from Government of Amazonas and; (v) US$ 0.08 million was from project beneficiaries. At the time of project cancellation, only US$ 0.11 million of the GEF grant had disbursed (approximately 2 percent of the original grant amount). The undisbursed amount of US$ 6.47 million was cancelled. The NCO does not report on the total project costs at closure or on how much of the expected counterpart contributions from government and beneficiaries had been provided.

The Project was approved by the Board on June 13, 2006 and was initially implemented by Secretariat of Biodiversity and Forests in the Ministry of Environment. Low capacity and commitment in the implementing agency, however, delayed the establishment of the project coordination unit. Once established the unit was chronically understaffed, culminating in the loss of the project coordinator in mid-2009. By the end of the third year, very few activities were implemented and less than 10% of the grant was disbursed. Consequently the grant was amended on March 26, 2010 to allow for the transfer of implementation responsibilities to the Brazilian Institute for the Environment and Renewable Natural Resources (IBAMA). The grant amendment became effective on June 11, 2010, but was never implemented because IBAMA had undergone an internal restructuring in the interim that resulted in a change in leadership and promised staff did not materialize. As a result technical cooperation subsidiary agreements with each of the States and the contract with the financial agent for execution of sub-projects, were never signed or implemented. The project was cancelled on March 26, 2012, 5 months before the original date of August 31, 2012.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Rated Substantial. The global environment objectives were relevant to country priorities identified national biodiversity and water resources strategies and the World Bank Country Assistance Strategies. Specifically the Global Environment Objectives are consistent with the National Biodiversity Policy's emphasis on the conservation and sustainable use of aquatic resources; and, the National Water Resources Policy and the National Forest Program's focus on the restoration of degraded forest in areas essential for the maintenance of aquatic and terrestrial biodiversity, and of water quantity and quality. The global environment objective is aligned with two of three long-term objectives specified in the Country Assistance Strategy in place at project appraisal: (i) improvement of water quality and water resources management; and (ii) sustainable management of land, forests, and biodiversity. The GEO remains relevant to the current Country Assistance Strategy's strategic objective for improving sustainable natural resource management and climate resilience. Finally the GEO is also consistent with the GEF Operational Program for Coastal, Marine, and Freshwater Ecosystems (OP2), the Operational Program for Integrated Land and Water Multiple Focal Area (OP9), and the GEF Strategic Priority #2 for the Biodiversity Focal Area (Mainstreaming Biodiversity in Production Landscapes and Sectors).

b. Relevance of Design:
Rated Substantial. The project activities were substantially relevant to the achievement of its global environment objective. The project was designed to pilot innovative bottom-up mechanisms for stakeholder involvement in the conservation and sustainable use of freshwater ecosystems. Project design included activities to diagnose the main threats to the to the integrity of freshwater ecosystems and the barriers to addressing them; support a participatory consensus building process that would establish a plan tor the conservation and management of aquatic resources; and, establish a governance strategy that would support long term implementation of this plan. Design also incorporated a decentralized and inter-sectoral management approach and measures to resolve conflict over the use of aquatic resources, both relevant features to achievement of the objective.


4. Achievement of Objectives (Efficacy) :

Not applicable for NCO.

5. Efficiency:

Not applicable for NCO.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Not applicable for NCO.

a. Outcome Rating: Not Applicable

7. Rationale for Risk to Development Outcome Rating:

Not applicable for NCO.

a. Risk to Development Outcome Rating: Not Applicable

8. Assessment of Bank Performance:

a. Quality at entry:
Quality at entry is rated moderately unsatisfactory. The Bank engaged in a lengthy preparation process that included an extensive participatory consultation process and it established close relationships with Ministry of Environment, state governments and other partners. Project design was informed by prior Bank projects implemented in Brazil. However, the NCO point to several shortcomings in project preparation and design:

  • The project's risk analysis was insufficient, and neither the Bank nor GEF review processes picked up on this weakness. Specifically, the Ministry of Environment's commitment and capacity to internalize the aquatic biodiversity conservation agenda within the institution was overestimated. The risks associated with government internal bureaucracy, its capacity and readiness to manage and implement a highly complex project and its level of commitment were not sufficiently evaluated.
  • The NCO concludes that the global environmental objective was over ambitious and unrealistic in light of the complex local environment, conflicting interests and multitude of pressures on the aquatic resources. And that the project would have benefitted from a less complex and more flexible structure.

Consequently, despite significant effort to work with the government to initiate project start up and assist with project restructuring, implementation never took off fully. The NCO notes that project performance was poor from the onset and worsened over time. Changes in the Bank team composition mid-way through the life of the project reduced the momentum and affected the ability of the Bank to provide continued support to the client.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
Not applicable for NCO.

Quality of Supervision Rating: Not Applicable

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
Government Performance is rated Unsatisfactory. The project was initially designed to be implemented by the Ministry of Environment as the Executing Agency through its Secretariat of Biodiversity and Forests. Other partners included the National Indigenous Foundation and the state environmental agencies of the states of Amazonas, Mato Grosso and Par. The National Biodiversity Commission, which includes representatives from key ministries, civil society organizations, and representatives of sectors that use biodiversity resources, was selected to serve as the Project Steering Committee.

The NCO reports that the Brazilian government has successfully promoted the biodiversity conservation agenda and had been very supportive of the project and its objectives. In practice, however, within the Ministry of Environment the theme of aquatic biodiversity conservation was not well developed and was not considered a priority. While engagement of actors at the State and local levels was strong, the lack of interest and commitment at the federal government level is reported to have resulted in limited ownership and engagement and weak coordination between federal and state levels during project preparation and implementation. Ultimately, the project did not have a champion at the federal level that was committed to implementing the project from the beginning and supporting the restructuring later on. Specifically:

  • Establishment of the Project Coordination Unit was delayed. Once established, the unit was chronically understaffed and suffered high turnover, including the eventually loss of the Project Coordinator.
  • Inaction at the federal level impeded the ability of state and local governments to retain technical staff.
  • Conditions of effectiveness of the amended Grant Agreement were not met. Attempts at modifying the project’s institutional and implementation arrangements, were not followed through by the government and remained at the planning stage. Subsidiary agreements on technical cooperation with each of the states and the contract with the financial agent for the execution of sub-projects were not signed prior to the effectiveness deadline of June 30, 2011.
After several unsuccessful efforts to address the above issues, the government eventually decided to proceed with the cancellation of the grant.

Government Performance Rating: Unsatisfactory

b. Implementing Agency Performance:
Not applicable for NCO

Implementing Agency Performance Rating: Not Applicable

Overall Borrower Performance Rating: Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The project appraisal document envisaged the development of elaborate monitoring and evaluation system.
Time bound targets were set forth for the establishment of each element of the proposed system. Provisions for monitoring and evaluation were included as part of component 4 project management. The project coordination unit was assigned responsibility monitoring, evaluating, and disseminating project results. A State Project Committee was also expected to monitor project execution and suggest necessary adjustments. The main elements of the M&E system were:

  • A project Implementation Monitoring System to monitor the implementation progress of project supported activities and their impact on freshwater ecosystems.
  • An information system on aquatic biodiversity and fishery statistics (SIBA) that would include ecosystem level indicators, such as area of flood plain with original vegetation, and also habitat and species-specific indicators. SIBA indicators were to be monitored through partnerships with universities, research institutions, NGOs, and local community organizations. SIBA would initially cover the Project target areas but would eventually expand to include other areas of the Amazon Basin.
  • The information collected was expected to be included in a geo-referenced database that would be accessible to the public. Information generated by the M&E system was also expected to be disseminated through bulletins, folders, radio, television, internet, and other events.
  • A plan for development of the baseline was to be established at an initial M&E seminar to be held during the first year of implementation. The development of the baseline would use a methodology that permits local community participation in the process.
  • The results of data analyses and baseline study reports would be disseminated on the Project’s web page and through seminars with local communities. At these seminars, participants would discuss the status of aquatic resources and propose and prioritize actions to solve the problems identified.
  • An external and independent mid-term evaluation was to be carried out in the beginning PY 4. And a final external and independent evaluation would be carried out at the end of the project, focusing on the same questions and indicators as the mid-term evaluation.

The following indicators were set forth in the PAD to measure project outcomes:
A proposal regarding institutional arrangements and processes for integrated management of aquatic resources discussed with the nine States of the Brazilian Amazon by project year 6 (PY6);
Action programs for integrated management of aquatic resources operating in three project target areas, and covering an area of about 290,845 km2, with about 32,941 km2 of freshwater ecosystems benefiting from the sustainable management of its natural resources, including biodiversity (PY6);
Demonstration activities (at least five for each of the three sub-basins) selected on the basis of participatory diagnostics by the end of PY 1 and implemented from the first trimester of PY2;
Capacity strengthened to support integrated management of aquatic resources in the following key institutions: (i) public institutions (Federal government, 9 State governments, and at least 9 municipal governments); 15 non-governmental organizations (representative entities of civil society, cooperatives, colonies and associations of fishermen, indigenous associations, and others); 90 multipliers and leaders; 15 special interest groups (women, youth, farmers, and fishermen); 45 schools, and 45 local communities by PY6;
Area of riparian forests recovered or under sustainable management by PY6;
Increase in the average size of three of the main fish species captured in the project area by PY6;
Conservation of aquatic biodiversity and water resources taken into account in decision making processes associated with new investments and development plans in three subbasins - measured by the frequency of consultation with relevant stakeholder groups;
A financial sustainability strategy developed and negotiated by PY6;
150 training and environmental education events offered to natural resources users, technicians, and decision-makers in the three participating sub-basins by PY6;
Mobilization and raising of awareness in 45 local communities, 45 schools, and 15 nongovernmental organizations for effective participation in integrated management of aquatic resources by PY6;
Public fora in support of integrated management stakeholders strengthened and/or created in sub-basins (three local and three state committees strengthened by PY6);
An Information System on Aquatic Biodiversity (SIBA) created and made available to the general public, beginning in PY2; and
Objectives and results of the Project disseminated (at least three seminars and three diagnostic documents by PY2, two international seminars, six regional seminars, two external evaluation reports, 15 progress reports, and two media campaigns by PY6).

Most of the proposed indicators are process indicators that monitor completion of activities but do not measure actual results and many of the proposed outcome indicators are not sufficiently defined to clearly indicate what constitutes their successful achievement.

b. M&E Implementation:
Not applicable for NCO.

a. M&E Utilization:
Not applicable for NCO.

M&E Quality Rating: Non-evaluable

11. Other Issues:

a. Safeguards:
Not applicable for NCO.

b. Fiduciary Compliance:
Not applicable for NCO.

c. Unintended Impacts (positive or negative):
Not applicable for NCO.

d. Other:
Not applicable for NCO.



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Not Rated
Not Applicable
 
Risk to Development Outcome:
Non-evaluable
Not Applicable
 
Bank Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Borrower Performance:
Unsatisfactory
Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The NCO draws the following lessons, which IEG has adapted:
  • Project effectiveness can be hampered by a lack of realism in defining project objectives and inadequate risk analysis. The project global environmental objective of “assur[ing] the conservation and sustainable use of the Brazilian Amazon freshwater biodiversity” was overambitious and unrealistic in light of the complex local environment characterized by conflicting interests and a multitude of pressures on such resources. The inadequate and insufficient risk analysis carried out for the project and the failure of the different review bodies and processes to pick up on this weakness hindered execution from project start. Risks analysis should be comprehensive and scrupulous particularly in operations with complex structure and/or piloting new approaches. Project design needs to be based on a more accurate assessment of the implementation capacity and readiness of government. In hindsight, the project would have benefited from a less complex and more flexible structure and mechanism to support the subprojects, to be more aligned with government execution and management capacity and level of experience, and to be more compatible with a small grant aimed at generation and dissemination of sub-regional experiences that could be scaled up.
  • Projects are more likely to be successful if the responsibility for project implementation and flow of funding are closer to those whom the project interventions are more likely to benefit. The project enjoyed much support at state and local levels, where most of the activities were supposed to happen, however the primary responsibility for leading project implementation was with the government at the federal level. This misalignment, coupled with weak institutional capacity, created a bottleneck and lack of engagement.
  • Remedial actions to poor project performance, including the decision of cancelling an operation need to be taken quickly to be cost-effective. It took five years and ten supervision missions to decide to cancel this operation. Despite costly efforts by both the Bank and Borrower to improve the project performance the project disbursed less than 10% and achieved very little on the ground.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The NCO is satisfactory on balance. It provides a candid assessment of the implementation challenges that resulted in project cancellation. But there are some gaps in reporting on the actual project costs and financing amounts provided at the time of cancellation. The amount of the GEF grant that disbursed is provided but it does not report on the amount of government and beneficiary counterpart contributions that were provided or report on the final project cost and breakdown by component.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions