Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Energy Efficiency Project


  
1. Project Data:   
ICR Review Date Posted:
06/27/2014   
Country:
Serbia
PROJ ID:
P075343
Appraisal
Actual
Project Name:
Energy Efficiency Project
Project Costs(US $M)
 55.0  51.39
L/C Number:
C3870, L7466
Loan/Credit (US $M)
 21.0  48.99
Sector Board:
Energy and Mining
Cofinancing (US $M)
 None  None
Cofinanciers:
Board Approval Date
  03/16/2004
 
 
Closing Date
12/11/2008 04/30/2013
Sector(s):
Energy efficiency in power sector (97%), Central government administration (3%)
Theme(s):
Pollution management and environmental health (29% - P) Climate change (29% - P) Urban services and housing for the poor (28% - P) Public expenditure financial management and procurement (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Mamundi G. Sri-Ram Aiyer
Robert Mark Lacey Christopher David Nelson IEGPS1

2. Project Objectives and Components:

a. Objectives:
According to the Development Credit Agreement: "The Objectives of the Project are to: (a) improve energy efficiency in heating buildings in order to make heating more affordable and to improve function and health environment of the users; and (b) reduce the local and global environmental impact of the use of dirty fuels for heating buildings in the Republic of Serbia".

The statement of objectives in the Project Appraisal Document (PAD, p.2) is almost identical: "the development objective of this project is to improve energy efficiency in heating buildings in order to make heating more affordable as well as improve the financial and health environment of the users. An important associated objective is to reduce the local and global environmental impact of the use of dirty fuels for heating buildings in Serbia.”

Additional Financing (AF) of US$30 million became effective in November 2008, to help Serbia go beyond the technology demonstration pilots, overcome several barriers to the development of the energy efficiency market, and implement supply and demand-side improvements. Although the project’s scope was expanded, there was no change to the development objectives.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
There were three components:
Component 1 (a), (Originally Component 1): Environmental improvement, heating system renovation, and energy efficiency improvement at the Clinical Center of Serbia (CCS) in Belgrade. (Cost estimate at appraisal US$ 7.47 million; at completion US$ 15.0 million including AF). Originally consisted of 3 sub components: (i) production of about 60 MW of heat, sanitary hot water, and steam, and 2 MW of power generation; (ii) modernization of building heat and substations and reconfiguration of the heat distribution systems; and (iii) energy efficient envelope and roof insulation and windows, and installation of thermostatic regulators.
Component 1 (b): Under the AF, Component 1 was expanded to include the Clinical Center of Nis (CCN), the second largest clinical center after Belgrade (AF US$ 5.94 million, actual cost US$6.40 million). Two coal fired boilers were replaced by a single gas fired boiler. Energy efficiency improvements were implemented in 19 buildings. The energy system was designed taking into account energy efficiency measures in all CCN buildings, which reduced the energy supply requirements for any given level of demand. This approach would demonstrate the benefits from coordinated supply and demand-side energy efficiency measures.
Component 2: Energy efficiency improvements in Social Service Buildings (Schools and Hospitals) across Serbia. (At appraisal US$ 12.36 million; at completion, including AF, US$ 26.41 million). It financed energy efficiency measures in selected schools and hospital buildings through tailored investment packages combining supply and demand-side improvements. They included automatic temperature control, roof/wall/piping/basement ceiling insulation, window repair or replacement, balancing and thermostatic valves, and improvements in lighting in schools by providing adequate and energy efficient luminaries. A total of 60-70 buildings including 17 primary schools with heated floor space of some 51,000 sq. meters and 12 hospital buildings with a total heated floor space of about 68,000 sq. meters were to be retrofitted. The expansion at AF set the stage for improvements in end-use efficiency, triggering the development of a market for energy efficiency services in the country.
Component 3: Technical Assistance. (At appraisal: US$ 0.66 million; at completion US$ 1.38 million including AF). Tasks included: (i) design and supervision of works in public buildings, covering design, technical specifications, bid document preparation, procurement, civil works implementation, and quality control; (ii) M&E including purchase of monitoring equipment (e.g., heat meters, temperature loggers, wind meters etc.). Social monitoring data were collected to measure the impact of the investments on the energy efficiency behavior of beneficiaries; (iii) development and implementation of a communication plan for the project; (iv) project preparation costs for consultants for energy auditing and investment proposal preparation; and (v) project financial management and auditing. At AF, technical assistance was expanded to cover (i) technical support, e.g., energy audits and monitoring, and external communication activities; and (ii) financial management, auditing, and incremental costs, covering office space and facilities for the Project Implementation Unit (PIU), Operation & Maintenance costs of equipment, vehicles, office supplies, communications costs, translation and office equipment.

At appraisal, there were unallocated costs of US$1.50 million and price and physical contingencies of US$5.22 million. The incremental operating cost was US$1.43 million at appraisal and US$2.2 million at closure.

The project under went three level 2 restructurings (on June 20, 2007, January 29, 2010, and October 20, 2011) involving some reallocation of funds within and between components and extensions of the closing date (see below).

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost: The final project cost was US$51.39 million (including an incremental operating cost of US$2.2 million), i.e., 7% below the appraisal estimate of US$55.0 million (including the AF). Although the final cost of Component 1 rose by 100% to US$ 15.0 million, and the cost of works at CCN by 7% compared with the AF estimate to US$ 6.40 million, the final cost of Component 2 dropped to US$ 26.41 million, or some 80% of the original estimate. Component 3 (TA), accounting for the smallest share of total project costs rose by 109% to US$ 1.38 million.
The doubling of the cost of Component 1 was due to (i) a rise in cost of labor, material and energy installations; (ii) depreciation of the exchange rate, which raised the cost of imported materials; and (iii) underestimated costs in the technical specifications due to a lack of experience in the implementation of large and complex demand and supply-side investments in the country. The specifications were based on preliminary designs and cost information was inadequate. (Nonetheless, because of higher than expected savings in operational expenditures as a result of the highly efficient heat delivery systems, the economic returns were still attractive – see Section 5 below).
Financing: Project financing consisted of the initial IDA Credit of US$ 21.0 million for a project of US$ 25.0 million, and Additional Financing (AF) of US$ 28.0 million (US$ 10.0 million in the form of a further IDA Credit and an IBRD Loan of US$ 18.0 million) for a total project cost of US$ 30.0 million with the-Borrower contribution)The AF-supported expansion utilized the original project's implementation arrangements, and was aimed at maximizing cost effectiveness and timeliness, and complement ongoing activities.
Borrower's Contribution: Under the initial project the Borrower's contribution was expected to be US$ 4.0 million equivalent, and under the AF an additional US$ 2.0 million equivalent. In the end, the Borrower's actual contribution was US$ 2.16 million equivalent, or 36% of the amount foreseen. In 2013, counterpart funding of US$ 2.0 million from the Central Government had not materialized by closure. In its place the Borrower made available, through the Ministry of Health (MOH) and the local government of the City of Belgrade, some funds to avoid a further reduction in the number of hospital buildings that were to benefit from improvements in energy efficiency, and for the gas pipeline to the CCS.
Dates: The project prior to AF was extended once from April 30, 2010 to October 31, 2011, in order to finalize the completion of heat supply rehabilitation works and energy efficiency improvements in the CCN and retrofit
selected public buildings with energy efficiency equipment. The post-AF project was extended twice. The first extension, from April 30, 2010 to October 31, 2011, was due to delays in Parliamentary approval of the AF, which meant it became effective 17 months after Board approval. The second extension, from October 31, 2011 to April 30, 2013, was due to procurement-related delays.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High


The project’s objectives are relevant to the environmental sustainability pillar of the Country Partnership Strategy for the Fiscal Years 2012-2015, which states that ‘lack of investment in the energy sector left Serbia with twin challenges: (i) ensuring that inadequate power supplies do not stifle the country’s efforts to improve competitiveness; and (ii) becoming more energy efficient, since the economy uses two times more energy per dollar of GDP than its neighbors in the Western Balkans and almost 2.5 times more than EU countries’.

The objectives were in line with the assistance program for the Fiscal Years 2003-2004 outlined in the August 2002 update of the Transitional Support Strategy (TSS) for the Federal Republic of Yugoslavia (FRY-Serbia and Montenegro) and reconfirmed in the February 2004 update.

The objectives were also directly relevant to the Government's goal of improving the economy’s competitiveness, for which serious infrastructure bottlenecks and inefficiencies had to be addressed. Energy imports were major factors in the trade and current account deficits, with the bulk of natural gas (85% of which was imported) used for heating (87%) rather than industry (12% of gas used for feedstock).

The objectives consider the needs of the energy and environment sectors, and their contribution to economic development. (The World Bank's Environment Department gave the project its 2007 Green Award for effectiveness and results).

b. Relevance of Design:
Substantial.

The project inputs focused on energy efficient retrofitting measures in prominent, highly visible public buildings, and the demonstration effect that would have to raise awareness of the importance of energy efficiency. The benefits are also meant to be understandable to the wider community and enhance confidence in further public expenditure commitments. The inputs included implementation arrangements involving several institutions towards achieving the necessary outputs, while also enabling public officials to gain practical experience in managing the installation and operation of energy efficiency investments in priority buildings. These were expected to contribute to the outcome of improving financial and health environment of users, besides reducing emissions of pollutants, leading to pubic approval to promote more systemic replication. Among the inputs was technical assistance financed for the first three years by the European Agency for Reconstruction, to help the Serbian Energy Efficiency Agency (SEEA) develop M&E capacity, training and communications activities for the project and follow activities, and for public buildings, assistance with design, procurement, contract administration and quality control. M&E measures built into the project served to assess progress by tracking specific outputs as implementation proceeded. These were based on detailed terms of reference for consultants (PAD, Annex 11). The design of the results framework tracks expected outcomes, reduction in emissions, energy savings of retrofitted buildings, increased end user satisfaction and improved health environments, expanded use of gas for heat, hot water and steam, number of buildings retrofitted etc. The results framework clearly links inputs, intermediate and final outputs and output indicators to outcomes, along with a fair assessment of potential risks and mitigation measures.


4. Achievement of Objectives (Efficacy) :

a. Improve energy efficiency in heating buildings in order to make heating more affordable and to improve function and health environment of the users: rated Substantial
In terms of outputs, 82 buildings were retrofitted (lower than the target of 111 buildings due to cost overruns), including 36 schools, 41 hospital buildings and 5 social care institutions. CCS had new gas-fired boilers with a total of 50 MW capacity and a central heating plant with 2+2 MW capacity were installed, rated at 93% efficiency. In CCN efficiency was 95%. Both values for efficiency exceeded the original targets. 129 energy efficiency audits were performed in schools, hospitals and social care buildings, exceeding the target of 70 to 80 energy audits. Except for CCS, in all other buildings measurement equipment was installed before the works, thus allowing for monitoring of energy savings. For CCS because of shortcomings in baseline data, the energy savings are based on those values that were measurable.
In terms of outcomes,, the average annual energy consumption in schools was 125 kwh/sq.m baseline (373, target 207); in hospitals 185 kwh/sq.m (baseline 395, target 231; in social buildings 175 kwh/sq.m (no baseline or target). Overall, energy consumption in these buildings decreased by about 60% according to monitored data.
Financial savings from energy efficiency improvements accrued to municipalities for schools, the state budget for hospitals (through the Health Insurance Fund), which made adjustments to the entities' energy budget as a result of the works. The estimate of savings from the new heat supply systems of the CCS and CCN is between €0.5 and €1.0 million pa. Of the total of 82 buildings,14 retrofitted buildings were connected to District Heating (DH), and it is assumed that the financial savings accrued to these public owned DH utilities. (End beneficiaries--schools and hospitals--were not obliged to repay investment costs). Improvements in heating efficiency were, therefore, measurable and significant, supporting the conclusion that more efficient heating equates with more affordable heating.
End user satisfaction increased by around 130% as measured through perception surveys including perspectives on improved functionality and health environment. End user awareness also increased as a result of the choice of public buildings for retrofitting such as schools, hospitals and social buildings. Awareness of energy efficiency works undertaken and the benefits of energy efficiency increased by an average of 240%. End users also confirmed that their application of energy efficiency measures at home increased by 90%. The methodology for the surveys was prepared by consultants in accordance with terms of reference issued by the Bank (PAD, Annex 11).

b. Reduce the local and global environmental impact of the use of dirty fuels for heating buildings in the Republic of Serbia: rated High
Emissions of local pollutants (SO2, NOx, ash, soot and solid particles) and greenhouse gases (CO2) produced at CCS and CCN were significantly reduced. Emissions of SO2, ash, soot and other solid particles were eliminated since both CCS and CCN now use natural gas instead of low quality coal for all their heat, hot water and steam (for example, CCS' SO2 emissions of 780 tons per annum (tpa), and CCN's SO2 emissions of 220 tpa both dropped to zero); CCS' NOx emissions dropped from 100 tpa to 50 tpa,; CCS' ash emissions of 130 tpa and CCN's ash emissions of 53 tpa both dropped to zero. Greenhouse gas CO2 emissions by CCS and CCN dropped from 48,000 tpa to 25,000 tpa, or by some 50%. The local and global environmental impact from use of dirty fuels for heating was, therefore reduced.

5. Efficiency:

Efficiency is rated Modest.
  • Cost overruns, limited capacity for procurement and complex supply-side energy efficiency measures in the CCS led to delays in implementation for 3 years after approval, until 2007. The Project Implementation Unit (PIU) also experienced a steep learning curve. Delays in Parliamentary approval of the Additional Financing (AF) delayed completion of retrofitting of two schools planned for the first phase, and of rehabilitation works at the CCS maternity hospital, as CCS had to wait for funds from the AF. Three extensions of the closing date were approved by the Bank. The initial project closed 18 months behind schedule, while the post-AF project closed three years behind schedule.
  • Project cost at completion was below the initial estimate (appraisal plus AF), but the number of buildings retrofitted was reduced from 111 to 82 due to cost overruns.
  • A key determinant of the return on Component 1 a (CCS) was the fuel substitution. Although the final cost of this component was US$ 15 million, some 100% above the initial estimate of US$ 7.47 million, by the end, the original fuels, mostly heavy fuel oil, and some light fuel oil and coal were all replaced by natural gas which costs much less. Therefore, the ex-post economic rate of return (ERR) for this component is some 33% compared with the ERR at appraisal of 32.4%.
  • For CCN, the ERR, estimated at appraisal at 13.3%, was finally estimated at some 23.9%, because over time, energy prices had increased. Again the CO2, SO2, and NOx emissions were reduced to a greater extent than in CCS, because -- besides the total rehabilitation of the heating plant -- all the distribution piping, heat sub stations and building envelopes were also retrofitted. Other tangible results are improvements in air quality and working conditions for end users.
  • For Component 2 (Schools and Hospitals), which accounts for over half the total project cost, the appraisal estimate of ERR was 13%, assuming constant energy prices in real terms, while, assuming energy price increases of 3 % pa (assumed difference in increase of prices between heavy fuel oil and gas), the ex post ERR is estimated at16%.

    a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
13%
100%
ICR estimate:
Yes
16%
100%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Based on the country's needs and sector priorities when the project was begun, and at completion, the Relevance of the Objectives is rated as High. Relevance of Design is Substantial, as inputs were logically linked to physical outputs and outcomes in energy efficiency, environment and health. Efficacy of the first objective -- to improve energy efficiency in heating buildings in order to make heating more affordable and to improve function and health environment of the users -- is rated substantial: the outcomes were fully achieved, but for 26% fewer buildings than originally planned. Efficacy of the second objective to reduce the local and global environmental impact of the use of dirty fuels for heating buildings - is rated high. Efficiency is rated modest: although the economic returns and environmental impacts were equal or superior to projections, there were serious delays in implementation and procurement in the both the initial and post-AF projects. Moreover, although the expanded project was implemented at below the expected final overall cost, cost overruns led to a smaller number of buildings benefitting from retrofitting. Overall outcome is assessed as moderately satisfactory

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The main risks to sustaining the outcomes are a decline in government commitment, reduced technical and financial viability of project activities, and a lowering of the capacities developed in the institutions, public and private, in energy efficiency and environmental activities. At project completion, the government (through the Ministry of Energy Development and Environmental Protection (MEDEP) began work with Germany’s Kreditanstalt für Wiederaufbau (KfW) and the Japan International Cooperation Agency (JICA) towards thermal rehabilitation of schools that were in the pipeline of the World Bank financed energy efficiency project. The sustainability of the outcomes is likely as MEDEP also confirmed its interest in scaling up efforts in energy efficiency in the buildings sector to: (i) reduce the fiscal deficit by decreasing energy related expenditures; (ii) mitigate the impact of needed energy prices; and (iii) renovate the building stock while moving to meet its national energy savings targets. MEDEP has requested World Bank technical assistance in the formulation, design and implementation of such a large scale energy efficiency program. Technical and Financial: Both CCS and CCN are operational, with the former being the most modern and environmentally friendly system in Serbia. According to the ICR (page 10), both systems are assisted by professional operators, qualified in modern and fully automatic control systems, and in emissions monitoring. The systems are profitable with revenue from energy sales able to cover costs of inputs and maintenance.
Institutional: Progress was made during the project on establishing the legal and regulatory framework to support the development of an energy efficient market in the buildings sector, such as an Energy Efficiency Law, amendments to the Construction Law to introduce standards for new buildings and renovations, and amendments to the Public Procurement Law to include energy efficiency criteria. The government has also started developing its Second National Energy Efficiency Action Plan (NEEAP) and energy efficiency indicators. Strengthened capacity at the Project Implementation Unit was developed with the aid of consultants, whose retention is key to ensuring that this is sustained.

In sum, since the technical, financial and institutional issues are being resolved with evident government commitment, the Risk to Development Outcome is rated as Low.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:
The Bank ensured that project objectives and parameters were consistent with the government's priorities, with the Bank's country assistance strategy, which continues to attach priority to energy efficiency and environment, and with South East European countries' strategy towards the creation of an Energy Community with a view to broadening the EU energy market. The US Agency for International Development (USAID), the European Agency for Reconstruction (EAR), KfW, the Swedish International Development Agency (SIDA), Norway and the European Bank for Reconstruction and Development (EBRD) were providing or planning to support investment and technical assistance for efficiency activities. The Serbian Energy Efficiency Agency (SEEA) was to leverage IDA support to strengthen donor partnerships to develop a comprehensive approach towards energy efficiency.
Prior to the project, the reform agenda for Serbia focused on the legal framework to enable commercialization, improve governance and increase competition within the sector. The 2004 Energy Law aimed to open the electricity market to competition, and gave independent producers access to the transmission system. It established the framework for the Energy Community Treaty in 2005, and Serbia formally joined the regional and European energy integration process. As explained in the PAD (p 20), the Bank ensured that the Government of Serbia's commitment and ownership of the project were clearly established.
The Bank played a key role in Serbia's energy sector agenda by guiding the relevant Ministry of Mining and Energy, (MOME), the PIU and the SEEA during preparation on the central issues of design – focusing on key relevant aspects, such as technical, environmental, social, financial, economic, procurement and financial management. The Bank adapted good practices from across the East European region. It also drew upon experience from implementation of other operations in Serbia, such as the energy pricing reform in the preceding Structural Adjustment Credit.
SEEA, a non-revenue earning agency, financially supported by the EAR, was given co-responsibility for implementing the project without assurance of sustained adequate funding. Neither the Borrower nor the Bank team prepared a contingency plan were EAR funding to cease, which in fact occurred (see Section 8b below).
In addition, the costs were underestimated in the initial project, attributed (in the ICR) to lack of experience in such complex systems. According to the project team, the preliminary costs were prepared by special consultants, who seem to have been less than rigorous than required, leading to serious underestimates.
M&E design was well designed, and described in the Project Implementation Plan (see Section 10a below). Provisions for safeguards and fiduciary complains were adequate - a financial management action plan was agreed with the borrower, with arrangements for frequent integrated fiduciary reviews (see Section 11 below).

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:
The Bank team visited the project approximately twice a year, with some 22-23 staff-weeks per year spent on supervision between FY 05 and FY 11. These visits focused on the development objectives and alerted the borrower about issues during implementation, and agreed on next steps and follow up actions, while also including them in Aide Memoires. However, the high turnover of Task Team Leaders(TTLs) during the project life at times rendered policy dialog and tracking progress on agreed actions difficult. The ICR (p 22) says there were five Task Team Leaders during the project period. According to the last task team, the reason was due to a reassignment in one case, and at the end, the current TTL who had some experience with "turn-around" of delayed projects was brought in from another region because of inordinate delays and large (US$ 22-23 million) undisbursed balances two years prior to the original closing date. The closing date was postponed and the project implementation speeded up with Bank staff assistance.

Important delays were also caused by the fact that construction permits in Serbia have been taking three times longer than in other neighboring countries, although this should have been allowed for by the task team. This was another factor contributing to much higher final costs than initially estimated.

The ICR (page 22) says that ratings in supervision reports of project performance in terms of development objectives and implementation management were reasonable--however, important delays and large cost overruns were not flagged. Supervision report ratings on procurement were Moderately Satisfactory or Satisfactory in the early years despite delays in evaluation of procurement contracts for consultants, which resulted in some slippages in procurement targets. The cost overruns and limited capacity for procurement and implementation of complex supply side energy efficiency measures in CCS led to delays in the initial project implementation until 2007 (the initial project's completion was set for October 2009). This underscores the frequent lack of realism with ratings in supervision reports. (The only exceptions seem to be the report of April 15, 2010 which rated implementation progress as moderately satisfactory, and that of May 19, 2011 which rated both development objectives and implementation progress as moderately satisfactory).

EAR’s financial support for SEEA ended by the third year of the agency’s operations. SEEA had to significantly adjust its salaries to be in line with the government’s compensation policies. This led to a significant disruption as most of its staff progressively left the agency. SEEA's responsibility for the project was taken over by the Ministry of Energy Development and Environmental Protection (MEDEP), which provided continuity.However, the lack of continued financing for SEEA as the implementing agency led to a loss of skills that had been built up over 5 years and had to be learned by MEDEP. This problem might beneficially have been flagged earlier by both the preparation and supervision teams in order to start identifying alternative sources of funding during the first year of implementation.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The ICR (page 23) reports that the government demonstrated strong commitment to the project from the start with enactment of legislation prior to the project, and taking steps, including legal and regulatory reforms, aimed at improving energy efficiency and the environment, as steps towards joining the EU Energy Integration Community. Institutional coordination across many Ministries for a project that was truly multi-sectoral (energy, health, education and social welfare) required major resource commitments, that were actually provided. MEDEP was effective in its coordinating role.

In October 2002, the government established the Serbia Energy Efficiency Agency (SEEA) to guide policy and implement urgently needed programs to increase energy efficiency. One of the initial goals of the project was to assist SEEA in developing its capacity to provide policy advice, design and execute programs that raise awareness of energy saving opportunities, and introduce incentives to help stakeholders act on these opportunities. In the event, external funding for SEEA was discontinued and no contingency plan was in place.
Long delays in Parliamentary approval of Additional Financing led to effectiveness 17 months after Board approval, adversely affecting the implementation schedule. The Project Implementation Unit (PIU) eas permitted to hire additional staff, which became necessary in 2010 to accelerate overall project implementation.

Delays in procurement prior to Additional Financing were attributed by the ICR to lack of experience with this type of project.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
Overall responsibility for implementation was under an inter-ministerial "Project Steering Committee" (PSC) chaired by the Ministry of Mining and Energy (MOME) and consisting of the representatives of eight different ministries. To carry out the day-to-day activities of project implementation, including operation of the Special Account and consolidation of Project Management Reports, a small Project Implementation Unit (PIU) was created under the supervision of the PSC.

The PIU’s capacity improved consistently during the 10 year implementation period. Its achievements include: (i) technical improvements to project design and activities that became part of "standard" technical specifications; (ii) social monitoring to evaluate progress measured by user satisfaction, energy efficiency awareness, and adequacy of working conditions; and (iii) highly organized and robust procedures for implementation of large scale projects (including preparation of energy audits with auditors, engineering with consultants, implementation with contractors, and supervision and monitoring.) The earlier performances of SEEA and that of MEDEP (which took SEEA’s functions) were satisfactory in both policy and institutional areas. The quality of M&E implementation and utilization was generally high (see Section 10b below). Safeguards compliance was considered satisfactory in supervision reports (see Section 11 a below). Other than procurement-related delays prior to Additional Financing, there were no fiduciary issues reported (see Section 11b below).

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
M&E design, agreed upon during project preparation, was described in the Project Implementation Plan prepared by the PIU, which was responsible for ongoing monitoring. Clinical Center staff, the PIU and SEEA were responsible for elements under their purview, with SEEA's tasks later undertaken by MEDEP.
The outcome indicators were appropriate for measuring progress towards achievement of the project's development objectives. They were realistic and useful for assessing progress towards specified quantified targets.
They included reduction of emission of local air pollutants and greenhouse gases produced by the Clinical Centers, quantified energy savings of retrofitted buildings and the cost of heat per square meter, and changes in end-user satisfaction, including improved functionality and health environments through perception surveys: Under Component 1, the Clinical Centers that shifted to gas for heat, hot water and steam, improvement in patient and other end-user comfort, air quality and outdoor air temperature and reduction in energy consumption were to be tracked; for Component 2, the number of buildings retrofitted; under Technical Assistance, progress in design, procurement and implementation of sub projects and quality of works executed, completion of annual communication plan and budget, number of local communities consulted and other outreach programs undertaken; and under the M&E program, satisfactory design, collection of data, analysis and utilization for feedback to policy makers and for refining communication plans for promoting further energy efficiency measures.

b. M&E Implementation:
M&E implementation was assisted by local consultants with some equipment acquired for a comprehensive M&E program. Participating schools and hospitals assisted in collection of data. Baseline data were to be collected as a requirement for participation in the project. Step by step monitoring, verification and evaluation by the PIU and consultants were central. The PIU sent quarterly progress and financial monitoring reports to the Bank. The data gathered were evaluated and served as inputs to decision making (e.g., for reallocation of funds) and to improve the technical specifications of sub-projects.
Social monitoring data were properly collected and evaluated on end-users' behavior, awareness and level of satisfaction, with baseline data collected before the retrofitting began.
M&E was implemented in considerable detail. There were some shortcomings in the monitoring of some emissions and of financial savings (which were managed by third parties), and sometimes because of nearby non-project works such as road construction and traffic rerouting, that affected measurements. It is not evident that these could have been foreseen or avoided.

a. M&E Utilization:
M&E data were consistently utilized for adjustments and corrective actions during implementaion, as well as to improve technical specifications. Thus, M&E results from implementation of the first phase of building retrofits were utilized to improve the design of energy efficiency measures in the expanded AF project, which led to a 10% increase on average in energy savings. The M&E data served as a basis for the Mid Term Review and for definition of key implementation issues requiring Government and Bank attention. Social monitoring data on end-users' behavior and level of satisfaction were used to identify and resolve unforeseen issues that arose after implementation (e.g., change of ventilation system for classrooms given that the upper sash window was not being well operated by users).

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:
The original Environmental Assessment category was "B" and this remained unchanged after the Additional Financing (AF). As well as the Environmental safeguard (OP 4.01), Involuntary Settlement and Land Acquisition (OP 4.12) was also provisionally triggered (PAD, p. 17). In the event, more detailed feasibility studies for the rehabilitation activities in the CCS and CCN confirmed that no acquisition of land or displacement of people would be required.

The project mostly involved rehabilitation works. According to the ICR (page 11), an Environmental Impact Assessment, including an Environmental Management Plan (EMP) for the CCS and CCN components was prepared, public consultations held and the EMP publicly disclosed. The EMP for the energy efficiency retrofits of buildings prepared during implementation was also used for activities under the AF. Rehabilitation of the CCS and CCCN required environmental protection measures during construction of the works, as well as provisions for emission controls in the design and operation of boilers. The energy efficiency retrofitting works involved improvements in heating systems, insulation, thermostatic valves, and other installations in public buildings.
The ICR reports that the PIU ensured that contractors complied with applicable health and safety regulations during execution. Regular environmental safeguards reviews were undertaken by the World Bank team, which: (i) assessed that works were executed with due care for environmental protection as well as for the users of public institutions; and (ii) recommended proper documentation with respect to safeguards control. “There were no significant deviations or waivers from the World Bank's safeguards policies. Compliance with World Bank environmental safeguards was considered ’Satisfactory‘ throughout implementation” (ICR, page 11).

b. Fiduciary Compliance:
Procurement. Beyond delays in the initial stages, there were, according to the ICR, no major procurement issues during implementation. Supervision report ratings on procurement were moderately satisfactory or satisfactory in the early years, despite delays in evaluation of procurement contracts for consultants, which resulted in some slippages in procurement targets. Procurement-related risks were generally identified appropriately and in a timely manner by the PIU, which continuously increased its procurement management capacity. There were no reported cases of mis-procurement.

Financial Management. The project used an automated accounting system, with quarterly financial management (FM) reports submitted within the deadlines to the World Bank. An Action Plan on FM, agreed upon with the Borrower based on frequent integrated fiduciary reviews, was implemented. The FM action plan contained specific deadlines and remedial actions for internal control weaknesses. The ICR reports that the PIU was adequately staffed for the FM function with an experienced and qualified specialist. External audit reports were satisfactory--the project team confirmed that they were all unqualified. The World Bank's FM specialist frequently carried out FM supervision missions to assess FM arrangements, which were rated 'Satisfactory' throughout project implementation.

c. Unintended Impacts (positive or negative):
Unintended positive outcomes included:
a. Technical capacity building in energy efficiency advanced in terms of engineering skills in project implementation, establishment of a professional pool of trained energy auditors in regional energy efficiency centers within 5 universities across the country, and the completion of 129 high quality preliminary and detailed audits to identify a final list of public buildings to be retrofitted--64 schools, 48 hospitals, and 17 social care institutions.
b. A Benefit Sharing Scheme was developed and Memoranda of Understanding signed with 10 local self-governments to ensure that a part of the financial savings was retained by end-suers. As of September 2013, one municipality has implemented the scheme.
c. A range of new consulting firms in energy efficiency monitoring and manufacturing energy efficiency equipment has sprung up as a result of the project, and the prospect of continued investment of this type.

d. Other:
None.



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
Although the relevance and efficacy ratings are high and substantial, efficiency is rated modest: while the economic returns and environmental impacts were equal or superior to projections, there were serious delays in implementation and procurement in the both the initial and post-AF projects. Moreover, although the expanded project was implemented at below the expected final overall cost, cost overruns led to a smaller number of buildings benefitting from retrofitting.  
Risk to Development Outcome:
Negligible to Low
Negligible to Low
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Satisfactory
Moderately Satisfactory
There were moderate shortcomings in government performance contributing to the important delays experienced in implementing both the initial and post-AF projects (see Section 8a above) 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The following three lessons are drawn by IEG:
  • Projects that seek to demonstrate a new approach in any country with new technologies are most effective when they obtain "buy in" from both government and end-users as was done in this case with surveys of user perceptions, and constant adaptation to meet their needs and expectations, while also serving to meet national goals.
  • Whenever a project includes implementing institutions financed by other external sources, the Bank should ensure that financing for such institutions will be available on a sustained basis, so as not to lose skills and learning developed during the initial implementation period.
  • Realistic reporting invariably helps in drawing the attention of borrower officials and Bank managers to problems before they become more serious, and risk achievement of project development outcomes. In this case, greater candor in supervision report ratings to cost over runs and delays in implementation at the earlier stages of the project would have drawn the attention of both client officials and Bank managers at an earlier stage than actually happened. Similarly early attention to finding alternative external financing for SEEA was called for.

14. Assessment Recommended?

Yes
Why?
This project would provide a good model for similar initiatives in this era when climate change is a priority global issue. The project performance having met all initial expectations is a good example from which there could be useful learning experiences for other government and Bank teams, and for other international financial institutions.

15. Comments on Quality of ICR:

The ICR is of generally good quality. The coverage of the achievement of objectives, and of details of implementation is quite thorough, and the ratings and lessons were evidence-based. The discussion could, however, have benefited from more emphasis on the achievement of the objectives as written, and less reliance on describing attainment of PDO indicators.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions