Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Energy Sector Technical Assistance Project


  
1. Project Data:   
ICR Review Date Posted:
06/28/2013   
Country:
Brazil
PROJ ID:
P076977
Appraisal
Actual
Project Name:
Energy Sector Technical Assistance Project
Project Costs(US $M)
 20.12  12.03
L/C Number:
L4708
Loan/Credit (US $M)
 12.1  7.7
Sector Board:
Energy and Mining
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  06/14/2003
 
 
Closing Date
12/31/2007 12/15/2010
Sector(s):
General energy sector (100%)
Theme(s):
Regulation and competition policy (50% - P) Other environment and natural resources management (25% - S) Other public sector governance (25% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Richard L. Berney
Robert Mark Lacey Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The project development objective as stated in the Loan Agreement (page 13) is: “to ensure sustainable implementation of the Borrower’s ongoing energy sector reform program.”

The project development objective as stated in the project Appraisal Document (PAD, page 2) is: “to help ensure sustainable implementation of the Government’s ongoing energy sector reform program, through technical assistance in specific areas, and by providing a mechanism for continuing dialogue with policy-makers about longer-term sector reform.”

This Review uses the statement of objectives in the PAD as it is more monitorable.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
A- Environmental Management (Appraisal US$2.34 million; Actual US$0.28 million)
This component was to support the mainstreaming of environmental concerns, and to complement institutional strengthening activities, covering, inter alia, environmental licensing, environmental change and social responsibility, assessment of issues and options in the area of climate change, monitoring and ex-post evaluation systems for environmental and social impacts of power projects, incentives for enhanced environmental and social responsibility, and institutional realignment and capacity building of professional cadres in environmental management.

B- Energy Market Development and Regulation (Appraisal US$2.61 million; Actual US$0.10 million)
This component included: provision of technical assistance on issues related to the efficiency of the energy markets (including links between electricity supply and the natural gas industry, concentration of ownership in the electricity generation industry, wholesale electricity market rules, retail level electricity market competition, links between wholesale and retail electricity markets, and barriers to international power trade), and on improving the regulatory activities for the electric sector (including links among the relevant federal regulatory agencies, and regulatory accounting and the electricity tariff review process).

C- Access to Energy for the Poor (Appraisal US$1.43 million; Actual US$0.49 million)
This component covered access and affordability of modern energy for poor consumers, including the development of: (a) comprehensive and consistent strategies for rural electrification, including policies and coordination of ongoing initiatives, baseline data collection, funding mechanisms, assessment and optimization of the development impacts and sustainability of the various electrification alternatives, and methodologies for monitoring and oversight of the implementation of new rural electrification programs; and (b) principles and eligibility rules to ensure that tariff discounts on electricity, liquid petroleum gas and natural gas are more effectively coordinated and better targeted towards the poor consumers on a sound economic basis.

D-Long-term Expansion Planning (Appraisal US$6.48 million; Actual US$4.72 million)
This component included long-term planning of the Borrower’s energy sector, covering: (a) preparation and updating of methodologies related to planning the expansion of the energy system, including integration of environmental and social concerns; (b) preparation and updating of selected inventories of river basins and feasibility of sites for electricity generation projects; and (c) the carrying out of a macro-location study on thermal plants to define the general areas in which those plants should be built, taking into account location of demand, fuel supply, land availability, and pollution issues.

E. Institutional Strengthening (Appraisal US$2.80 million; Actual US$4.94 million)
This component covered technical assistance and training (as required) for: (a) the restructuring and institutional strengthening of the Ministry of Mining and Energy (MME), mainly in the areas of integrated planning, international energy integration, universal access to energy, energy efficiency, alternative technologies and technology utilization, mines and metallurgy, and oil and natural gas; and (b) the improvement of the institutional structures that support the National Energy Policy Council (ConselhoNacional de PolíticaEnergética) and the Borrower’s Energy Development Fund (Conta de DesenvolvimentoEnergético).

Project Management (Appraisal US$1.13 million; Actual US$1.50 million)

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Cost: The total project cost at appraisal was estimated at US$20.1 million, of which the Bank was to finance US$12.1 million and the Government of Brazil (GOB) was to finance US$8.0 million. Actual cost for the components financed during the period between loan effectiveness and loan closing was US$12.03 million.

Financing: The Government considered the technical assistance activities supported under this project to be so time sensitive and important to its electricity sector development program that it implemented and financed many of the technical assistance activities originally proposed for the project with BR$37.5 million (equivalent to US$22.1 million) of its own resources during the 16 month period between Board approval and actual project effectiveness. Consequently of the Bank Loan, US$7.7 million was disbursed and US$4.4 million was cancelled.

Borrower’s Contribution: After the project became effective, the Government provided US$4.4 million equivalent to the project financing.

Dates: There was a 15 month delay in loan effectiveness. The main cause for this was the lack of continuity of in leadership at the beginning of President Lula da Silva’s first administration, which delayed completion of the local effectiveness requirements. The project closing date was extended three times for a total of three years, (from December 2007 to December 2010). The first one year extension was to compensate for the delays in achieving effectiveness. The second extension was to compensate for delays in implementation related to Government-imposed monetary policy restrictions on the transfer of funds to Brazil, including those from the World Bank to Government accounts. The third extension was to allow implementation of approved project activities that had been delayed due to Government budget cycle problems (see section 9a below).


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High: Brazil suffered an energy supply crisis in 2001, caused by low levels of investment and an unusually dry period. The crisis illustrated the adverse macroeconomic impact of the failure to implement energy sector reforms in a timely fashion. The Bank responded to the energy crisis with three loans: a quick disbursing Energy Sector Reform Loan, a Private-Public Infrastructure Advisory Facility program, and this longer term technical assistance project which was aimed at supporting sustainable implementation of the Government’s revised energy sector policy. The project objective is as relevant to Bank strategy for Brazil today as it was at appraisal. The 2008 Country Strategy Paper (page 12), states that improving Brazil’s competitiveness is one of the three primary areas of Bank group support, and within this context, Bank support is to focus, inter alia, on improving the planning and regulatory framework for infrastructure, expanding public-private partnerships, and “just in time” technical assistance for complex issues, such as hydropower development.

b. Relevance of Design:
High: The project results chain included the main inputs and activities that would seem necessary to achieve the project’s expected outputs and outcomes. All the studies identified in each of the project components focused on specific development objectives. Each project component had a specific development objective, and the components as a whole were designed to complement each other in their content and main objectives. For instance, the environmental management component supported revision of environmental licensing procedures, which were critical to the revised process of long term hydropower development planning. Establishment of a timely and transparent environmental licensing process and provision of incentives that promote implementation of environmental and social action plans would enable the Government to balance expansion and environment goals and, by eliminating licensing uncertainty, would encourage private investment in new energy projects. And management strengthening supported the creation of a new public agency for energy planning, financed a comprehensive Government Information System for the key energy sector indicators, and trained Ministry of Mines and Energy staff in the development and updating of the long term energy sector plans.

Close coordination on the technical assistance and studies implemented under the project facilitated meeting the objective of providing a mechanism for continuing dialogue with policy-makers about longer-term sector reform.


4. Achievement of Objectives (Efficacy) :

1. The degree of achievement of the project development objective -- to help ensure sustainable implementation of the Government’s ongoing energy sector reform programis assessed as substantial. The outputs and outcomes of the specific technical assistance areas are discussed below.

A: Environmental Management
Outputs: To help ensure long term environmental sustainability of power sector development, the project financed studies focused on evaluating environmental licensing processes and procedures. These studies undertook a detailed diagnosis of the Environment Licensing Process for hydropower plants, pointing out major causes of delays and institutional conflicts and providing a series of policy and procedural recommendations.
Outcomes: In a note transmitted to the Bank on March 2011, the Government acknowledged that the environmental studies financed by the bank had provided direct support to the Environmental law (7397/2008), three Decrees (No. 6.848, in May, 2009, No. 7.154, in April, 2010, and No. 7.342, in October, 2010) and two subsequent Ministerial Decisions, which focused on establishing a clear processes for environmental licensing of power sector projects.
As a direct result of these environmental Decrees and decisions, the environmental and social impacts of power projects are now assessed at the project concept stage, at which time the appropriate actions plans to avoid, mitigate or compensate impacts are to be developed. This has greatly improved the environmental licensing processes for hydropower plants and transmission lines, as well as on the environmental management of areas surrounding artificial reservoirs. One indication of the success of these new procedures is that the first environmental licenses awarded in twenty years were awarded in 2007 and 2008 to the two large Rio Madeira hydro power plants. According to the ICR, these changes in the environmental framework, along with an incentives program for alternative energy resources established in the Government’s Programa de Incentivo a Fontes Alternativas de Energia and in the auction-based national programs for renewable energy-based power generation investments currently under implementation, have also helped to encourage the use of non-hydroelectric renewable energy sources.

B: Market Development and Regulation
B-i. Evaluation of auction strategies for allocation of concessions for the two Rio Madeira hydroelectric projects:
Output: According to the ICR, these studies looked at private sector participation in the development of new energy sources and evaluated the strengths and weakness of competitive bidding processes for concession allocation of new hydroelectric projects compared to the then current practice of directly negotiated contracts with one company.
Outcome: The studies provided sufficient technical details to enable the Government to shift its project allocation strategy from noncompetitive negotiated contracts, to competitive international bidding. The subsequent competitive bidding auction process was used for two major hydroelectric projects, with significant cost savings. For the 3,150 MW Santo Antonio hydropower plant, the cost was reduced from the original single source offer of BR$140 per MWh to the auction price of BR$78/MWh, while for the 3,500 MW Jiraú hydropower plant the costs estimate declined from BR$120/MWh with the single source offer to BR$76/MWh from the winning consortium auction bid. The ICR estimates that the net present value of total savings from these two plants is approximately US$12 billion over their 50-year lives. The success of this competitive approach to financing new power projects convinced the Government that it should be used for all future hydro projects. This success has provided the Government with a sustainable auction strategy that has been shown to minimize the cost of new hydroelectric generation.
B-ii Transmission network optimization study
Output: The project financed a (Sistema de Medição Fasorial) evaluation study which included the testing of a new Phasor technology for the Ultra High Voltage (1000 V) transmission system (Sistema de Medição Fasorial), which is the high voltage network that evacuates power from the Rio Madeira hydroelectric projects.
Outcome: The study and testing program led to the decision to adopt the new Phasor technology for optimizing the operation and dispatch of the power in the main high voltage transmission grid. The cost of implementing the new optimization system is expected to be about US$3 million, and is expected to free-up an additional transmission capacity of around 1,000 MW, which represents a net savings of approximately US$4.8 billion over a 35-year lifetime. While one cannot allocate all (or any given percentage) of this saving to the study financed under this project, the study was instrumental in identifying the new investment that is expected to generate this benefit.
B-iii Smaller studies covering regulations for decentralized private based power generation in isolated areas, competition in the secondary natural gas market, and private sector power generation with small hydro power plants and biomass plants.
Outcome: The ICR states that these studies were completed and served to inform the reconsideration of aspects of the regulatory framework, in particular with a view to removing key regulatory bottlenecks for increasing private sector participation and competition in these areas.

C: Access to Energy by the poor
Outputs: The project financed six of the nine original project studies, the other three having been funded by the Government before the project became effective. The project-financed studies were designed to help improve aspects of Brazil's national rural electrification program, Luz paraTodos (Electricity for All). One study evaluated electricity tariff policies and subsidies for low-income households, and four identified the locations of 28 isolated Afro-descendent villages (Quilombos). A sixth study evaluated the adjustments in legislation needed to implement alternative energy generation projects in isolated communities in the Amazon.
Outcomes: Project-financed studies contributed to the design of funding mechanisms and affordable lifeline tariffs for poor communities that are now in place, based on an equitable threshold for lifeline rates. The identification and location of the isolated Afro-descendent villages (Quilombos), enabled the Government to include them in the Government’s Luz para Todos program.

D: Long-term energy sector expansion planning
Output: The project supported the development of 10 and 30 year Long Term Expansion Plans, which are now updated annually and every five years, respectively.
Outcome: Until the energy sector reforms in 1998 most power expansion planning was undertaken by ELETROBRAS, the major power utility in Brazil. With the progressive reduction in the mandate of ELETROBRAS in 1998 and 1999, planning expertise was largely dispersed, without a corresponding increase in the capacity of the Secretariat of Energy within the Ministry of Mines and Energy. The result was that there had not been any formal energy planning in Brazil since 1998. In 2002, before the project, energy sector policy making was fragmented and reforms were incomplete. Implementation of policy was also fragmented due to institutional gaps and the absence of a planning agency. The project supported the development of rolling 10 and 30 year framework expansion plans, which are now available on the internet to guide private sector investment proposals. The ICR reports that, by the time of project completion, energy and environment related policies, regulations and procedures were designed and implemented in a coordinated fashion by Government ministries, agencies and regulators, and that they are effective in fostering increased private participation.

E: Management Strengthening
The project strengthened the policy-making capacity of the Ministry of Mines and Energy through a series of studies and the provision of senior advisors and Ministry staff training programs in developing and updating the long term plans. It supported the creation of a new public agency for energy planning, the Institute for the Study of Energy. It also financed a comprehensive Government Information System for the key energy sector indicators.


2. The degree of achievement of the project development objective -- providing a mechanism for continuing dialogue with policy-makers about longer-term sector reform -- is assessed as substantial. The ICR reports that, on the strength of the outcomes of the studies and the policy dialogue that they engendered, the Government decided to implement a follow-up $50 million energy technical assistance project.

5. Efficiency:

The savings generated through the shift from direct negotiated power projects to auction based power projects for the two first auction based hydroelectric projects, estimated at US$12 billion over their 50-year lives was many times greater than the cost of this technical assistance project (US$12 million, of which the Bank loan was US$7.7 million) However, there were several shortcomings in the implementation of the project (described below) which had a negative impact on the efficiency with which project resources were used.

The project was implemented over a six year period, instead of the appraisal estimate of 4.5 years, and project management costs also increased by about a third. Project implementation started very slowly after effectiveness. In 2006 the Bank supervision team downgraded achievement of Development Objectives to moderately unsatisfactory, and Implementation Progress to unsatisfactory. As progress was made in the implementation of the studies financed by the project, Achievement of Development objectives returned to satisfactory by the end of 2007, and remained so for the remainder of the project. Implementation Progress returned to Moderately Satisfactory by early 2007 and remained so until project closure. Financial management remained unsatisfactory until close to project closure, and new plans had to be established during the last two supervision missions.

The Government's freeze on all public service recruitment affected the number and qualifications of the staff that the Project Implementation Unit could hire for much of its early years. This had a negative effect on project implementation schedules (see section 9b below)

The project suffered additional hurdles related to extension procedures for the three approved project closing date extensions. Repeated uncertainty about whether extensions would be granted by the Bank generated negative effects, which were exacerbated by government budget processing procedures (see Sections 8b and 9a below for details). As a result, in the final one year extension period, the supplementary expenditure authorization came so late that there was insufficient time to bid and complete new activities, and almost none of the still available Bank funds was spent.

Efficiency is rated as Modest.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Relevance of both objectives and design was high. The efficacy of both objectives is assessed as substantial. Policies, programs and procedures that were developed with the project’s support have been adopted as standard practice by the Government. The policy making and implementation capacity of the Ministry of Mines and Energy has been strengthened. The continuation of policy dialogue has been assured, and the Government has requested a follow-up bank-supported technical assistance project for the energy sector. Efficiency, however, is rated modest. Tardiness and procedural complications on both Bank and Borrower sides led to the Government financing many of the activities intended to be funded by the project, and to considerable delays in implementation of the rest. Overall outcome is assessed as moderately satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

For a Technical Assistance project such as this one, the risks are that of the Government will fail to utilize the information and processes that have been developed by the project studies or to follow through on the policies that have been developed in the technical assistance activities. In this case, however, development outcomes are likely to be sustained:
  • The project’s innovative practice of taking a systematic approach to developing studies to support policies and to inform policy makers when assessing alternatives to removing regulatory bottlenecks appears likely to be adopted as a permanent approach by the Ministry of Mines and Energy and associated agencies, such as the Federal Electricity Regulatory Agency.
  • The aim of making modern energy sources available and affordable to the poor is likely to be sustained by the renewed commitment of the current administration to the second phase of the “Luz para Todos,” program, which will increase access to energy services for poor populations.The risk that the information provided by the project's studies on the location of Qilombos communities that has now been incorporated into the program's data base, will not be used by the program is low.
  • TFederal environmental laws and related government decrees and decisions have been promulgated. They have established their benefits in mainstreaming environmental concerns in energy planning and project implementation, and there is a low to negligible risk that this legal and regulatory framework would be either repealed or significantly weakened.
  • Institutional strengthening of the Ministry and Mines and Energy -- in particular, the creation and reinforcement of the Energy Planning Agency and the Secretariat of Planning and Development -- is unlikely to be reversed.
  • The system-wide phasor measurement system to optimize the development and planning of the high voltage transmission system has adopted as one of National System Operator’s major development activities. Its full Implementation is part of the Ministry’s new project proposal to the Bank, which would reduce even further the already low technical and political risk to this development outcome.

    a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:
The justification for the project was sound, and extensive preparation work had been completed under the previous Private-Public Infrastructure Advisory Facility loan, including agreement with the Borrower on the studies to be carried out. However, the project’s preparation left several issues unresolved. The Ministry of Mining and Energy lacked experience in implementing a Bank-financed project. For this reason, the Bank rated procurement and financial management as high-risk. The only action that the appraisal proposed to mitigate this risk was that the project launch would include Bank specialists in procurement, financial management, and legal responsibilities. In addition, the Project Implementation Unit (PIU), under the Executive Secretary of the Ministry, was not going to be “fully operational” by the time of loan signing, and its full staffing was not a condition of effectiveness. There were difficulties in allocating adequate staff to the Unit associated with internal Ministry procedures. These difficulties were not anticipated. Consequently, the PIU was still under-staffed and inadequately trained at effectiveness, although effectiveness had been delayed by 15 months for other reasons (the main one being lack of continuity in leadership at the Ministry of Energy and Mines at the beginning of President Lula de Silva's first administration). The lack of PIU staff’s experience with Bank procedures meant that they were initially unable to produce terms of reference for complex issues, which delayed several activities. The preparation team did not anticipate the issues related to Government resource allocation procedures in this period of extraordinary budget constraints. This was a further cause of considerable implementation delays.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
Project implementation started very slowly after effectiveness, and by mid-2006 the Bank supervision team had downgraded implementation progress to Unsatisfactory and achievement of Development Objectives to Moderately Unsatisfactory.

The project suffered additional hurdles related to extension procedures for the three approved project closing date extensions. Repeated uncertainty about whether extensions would be granted by the Bank in combination with rigid government budget requirements created severe procurement complications. Specifically, the PIU had to hold up all new procurement processing until its expenditures were approved in the National Budget. And the project expenditures could not be put into its next year’s National Budget before the extension of the project closing date had been approved by the Bank. This sequential process resulted in an inefficient “stop and go” mode of operation that substantially increased implementation delays. The problem was exacerbated by the Bank consistently postponing the decision to extend the project closing date until almost the last moment.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
While the Government demonstrated strong ownership of, and commitment to, the project’s development objectives, its commitment to attaining these objectives through the utilization of foreign technical advisors for reviewing and evaluating sensitive policy matters fluctuated considerably over the project implementation period.

The Government considered the implementation of many of this project’s activities to be of such importance that it allocated US$14 million equivalent from the National Budget 2004 and 2005 for activates originally included in the project design, before the project became effective. However, continuity of funding became a problem. Government entities need to receive an allocation of funds from the annual National Budget process before they are allowed make financial commitments, and the Government usually only makes funds available for Bank projects when it has received commitments for reimbursements from the Bank. Subsequently, allocations from the Government budget were limited to those available for reimbursement by the Bank, as was the practice with all Bank financed projects. As noted in Section 8b, this process of prior allocation through the National Budget began to create a problem of mismatch of timing when extensions to the project closing dates were requested and when the Bank did not approve the extension before the national Budget was approved., As a result, in the final one year extension period, the supplementary expenditure authorization came so late that there was insufficient time to bid and complete new projects, and almost none of the still available the Bank funds were spent.

The project became effective at a time when the national treasury was blocking the transfer of all international currency into the Brazilian economy, as part of its austerity policy. This policy included blocking the Government’s New York account from receiving transfers from all World Bank loans. As part of its austerity measures, it also froze all public service recruitment for several years.

Government support for the project fluctuated. While Government agencies provided strong support for the project during preparation, after the new Government was elected in 2003, changes in the project’s political and administrative stakeholders led to widely varying periods of support throughout implementation. Lack of continuity of leadership within the Ministry of Mines and Energy at the beginning of President Lula da Silva’s first administration delayed completion of the local effectiveness procedures for 15 months. After the success of the bidding process for the two hydroelectric dams, the Government began to appreciate the potential benefits of project activities, and provided considerably more support for the PIU. Support again strengthened in the last year of project implementation when a second technical assistance loan was being discussed.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
The implementing agencies (primarily the Ministry of Mining and Energy and the Ministry of Planning) worked closely with Bank staff in designing the studies and support activities included in the project plan. They showed a strong fundamental commitment to the implementation of the project’s objectives, as evidenced by the fact that they used their own budgetary resources (through the national budget allocations) to finance project activities during the period before effectiveness and during the following period before fund transfer issues were resolved.

The Project Implementation Unit (PIU) was housed in the Ministry of Mining and Energy (MME), and many of its staff were transferred from other parts of the Ministry. However, restrictions on public service recruitment, which was frozen for several years, affected the number and qualifications of PIU staff. Since the number was frequently insufficient, remaining staff were over loaded. One result was that initially the Unit was inefficient in preparing and transmitting reimbursement requests, and was unable to produce terms of reference for several particularly complex subjects, which delayed several activities that the Ministry considered of high importance.

After the midterm review in 2007, an action plan to improve the PIU’s financial management and procurement capabilities was implemented. PIU staff was supplemented with long term consultants on three year contracts. However these consultants also lacked experience with management of Bank projects in general, and with the detailed requirements of Bank procurement and reporting requirements. The Bank subsequently provided additional training, and things began to improve. The Implementation Progress rating returned to moderately satisfactory by late 2007.

There were also several financial management problems (see Section 11b below). As a result, the project’s financial management performance was rated as moderately unsatisfactory, and the risk as substantial. However, with renewed Government support of the project in 2009, financial management improvements were observed following implementation of the action plans agreed upon during the last two supervision missions.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The outcome indicators, while relevant to the project objectives, were not well designed for an effective evaluation of project outcomes. Indicators such as “no more rationing” and “higher percentage of poor having access [to] modern energy services” are so broad, and are affected by so many activities beyond those of the project, that the outcomes cannot be fully attributable to project outputs. In this context other indicators (such as the success of action plans to avoid, mitigate or compensate for environmental impacts and that this compensation should be granted in a timely and transparent manner, and whether energy and environmental-related policies, regulations and procedures are designed and implemented in a coordinated fashion by Government agencies) would have been extremely difficult to evaluate, even if they could have been associated with specific project activities. To compensate for this difficulty, the project design included 47 output indicators associated with the project’s specific individual technical assistance activities. Responsibility for the M&E system was given to the Ministry of Mines and Energy through the Project Implementation Unit (PIU).

b. M&E Implementation:
The design of the project output indicators used in monitoring and evaluation was based on a set of assumptions, including, most importantly no delay in the achievement of the effectiveness date, and no change in the detailed list of numerous activities to be supported by the Project. When project effectiveness was delayed by 15 months, the Government financed a large number of the project’s proposed studies with its own funds. As a result, many of the 47 output indicators related to specific studies were no longer relevant for assessing the project‘s implementation progress. However, the indicators were not updated to reflect these changes. This contributed to the PIU’s inconsistent gathering of M&E data.

a. M&E Utilization:
Except for keeping track of the implementation progress for the studies financed under the project, M&E was essentially unutilized during project implementation. However, data on M&E indicators was collected in 2009-2010, during the last stages of project completion. This provided a basis for assessing which activities had been completed, and, in some cases, what improvements could be associated with (but not necessarily attributable to) these project activities.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
No safeguards policies were triggered by this Category “C” technical assistance project.

b. Fiduciary Compliance:
Several Financial Management problems arose during project implementation. These included weaknesses in internal controls; in the financial information management system; in meeting Bank reporting requirements; and auditors’ opinions were qualified on the Statement of Payments and Statement of Expenditures related to procurement of consultant services. In addition, the ICR reports (page 7) that the timeliness and adequacy of the Project Implementation Unit management’s corrective actions to address the auditors’ recommendations was less than desired. A plan to improve financial management, which aimed at strengthening staffing, budgeting, internal and external controls and reporting was agreed during the mid-term review. However, little progress was made and another action plan had to be agreed upon during the project team’s last two supervision missions. The ICR reports that this second plan was well implemented and financial management did improve. At project closing the Bank considered the Borrower’s financial management and disbursement arrangements and its compliance with the financial management covenants set out in the legal agreement, to be satisfactory. There was no misprocurement.

c. Unintended Impacts (positive or negative):
none

d. Other:
none



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
Relevance of both objectives and design was high. The efficacy of both objectives is assessed as substantial. Efficiency, however, is rated modest.  
Risk to Development Outcome:
Negligible to Low
Negligible to Low
 
Bank Performance:
Moderately Satisfactory
Moderately Unsatisfactory
There were several shortcoming in the Quality at Entry. The inability of the Bank to make timely decisions about project extension created major problems with implementation continuity. See section 8a.  
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The following lessons are taken from the ICR with some adaptation of language:
  • TA projects that are designed to support the requirements of a Government that is facing quickly evolving challenges require flexibility to include additional studies if they are to remain relevant. The flexibility to add new activities was critical to assuring that this project remained relevant to the evolving needs of the Ministry of Mines and Energy.
  • Trade-offs can arise between implementation of project activities according to the original schedule, and the need to take more time to ensure adequate discussions and consultations. When this occurs, prevalence of the latter would be needed in order to preserve ownership. These consultations were of particular importance for the successful results obtained in this project, particularly as needs of the Borrower evolved over time.
  • In Brazil, the timing of extensions of project closing dates needs to be coordinated with the Government’s budget process. When an extension is deemed to be necessary, the Bank needs to act expeditiously to assure that project implementation can continue in a timely, effective manner.
  • Extensive early training on Bank practices needs to be provided to ensure sufficient fiduciary capacity of the Project Implementation Unit. Attention to the training requirements came only at the mid-term review.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

This is a good, results oriented ICR, and with the exception of the excessive number of generic lessons, was fully consistent with the Bank guidelines. It clearly identified which outcome results could, and which could not, be attributed to the project. The evidence and the analysis are clearly presented, and the lessons were internally consistent. Some acronyms were not listed or defined. Some of the numerical presentations caused some confusion.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions