Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Energy Efficiency Development Policy Loan


  
1. Project Data:   
ICR Review Date Posted:
04/30/2014   
Country:
Poland
Is this review for a Programmatic Series?
 No
First Project ID:
P115426
Appraisal
Actual
Project Name:
Energy Efficiency Development Policy Loan
Project Costs(US $M)
 1,114.5  1,077.8
L/C Number:
Loan/Credit (US $M)
 1,114.5  1,077.8
Sector Board:
Energy and Mining
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  06/07/2011
 
 
Closing Date
03/31/2012 03/31/2012
Sector(s):
Energy efficiency in power sector (83%), Other Renewable Energy (17%)
Theme(s):
Climate change (92%) Pollution management and environmental health (8%)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Nestor Ntungwanayo
Kristin Hallberg Christopher David Nelson IEGPS1

2. Project Objectives and Components:

a. Objectives:

The loan objective was to support the Government’s program to decrease emissions through accelerating energy efficiency and targeted renewable energy interventions [Program Document (PD)], p. 22. The loan objective was not stated in the loan agreement, which instead listed the prior actions. As prior, medium and long-term actions were identified under each of the policy areas developed under Section 2c below, this review will use the policy areas as sub-objectives to assess achievement of the loan objective.

b. If this is a single DPL operation (not part of a series), were the project objectives/key associated outcome targets revised during implementation?
No

c. Policy Areas:

Policy Area 1: Develop a supporting legal framework for energy efficiency strategy: Reforms were to bring changes to the country's legal and regulatory framework of the energy sector, and to promote energy efficiency, notably by aiming at meeting European Union (EU) targets on energy efficiency in a quasi market-based manner, and by identifying additional actions over the medium term.

Policy Area 2: Decrease supply-side energy use: Reforms in this area aimed to increase the use of cogeneration for the production of electric power and heat, notably by accelerating construction of new small-medium sized cogeneration capacity, and identifying additional actions over the medium term as well.

Policy Area 3: Improve demand-side energy efficiency: Improvements in this area consisted in deploying smart grids and meters to enhance energy efficiency, accelerate demand-side management, and support the increased use of renewable energy resources. The reforms goal under this heading was to foster investment in energy savings devices, and in identifying additional actions over the medium term.

Policy Area 4: Renewable energy action plan: The reform agenda under this loan included the design of a renewable energy program to be launched right away, and to continue over the medium term. The program was to focus on wind power, biogas and biomass.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

This operation was a single tranche development policy loan approved on 06/07/2011 in the amount of $1,114.5 million. The loan had medium and long-term targets established, in case the Government wished to convert this program into a programmable series of development policy loans (DPLs). The loan was made effective on 07/05/2011 and an amount of $1,077.79 million was disbursed or 97 percent of approved amount, due to the euro appreciation against the dollar between the project approval and the disbursement period.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

High

Despite remarkable resilience to the global and Euro-zone financial crisis, Poland financial situation started to worsen in 2010. The fiscal deficit increased from 7.4 percent of GDP in 2009 to 7.9 percent of GDP in 2010, compared to the target of 6.9 percent of GDP, and a Eurozone goal of 3 percent of GDP. Public debt also jumped from 50.9 percent of GDP in 2009 to 54.8 percent of GDP in 2010.

With the objective of targeting EU convergence in terms of environment and climate policies, Poland adopted in November 2009 its Energy Policy until 2030. Selected key objectives of Polish energy policy were to: (i) improve energy efficiency; (ii) enhance the security of fuel and energy supplies; (iii) develop the use of renewable energy sources, including biofuels; and (iv) reduce the environmental impact of the power industry. In 2011, Poland faced the dilemma of reducing harmful emissions while maintaining its focus on economic growth to converge with EU-15 in a constrained fiscal environment.

The overarching goal of the 2009 Country Partnership Strategy (CPS) was to support Poland’s convergence towards EU living standards. The proposed Energy Efficiency DPL was an anchor for the strategic dialogue between the Bank and the Polish authorities, in the specific area of regional and global public goods. In 2011, Poland was in need of budget support to address treasury concerns, and a partner to help shepherding the energy reform in its quest to align with the EU energy efficiency standards.

b. Relevance of Design:

Modest

In the Bank's and the Borrower's perspective, this was designed as a one-off DPL. There was an ambiguity in the statement of objective of this operation, stemming from a situation whereby you have a PD that refers to the program objective (page i) and to the current operation objective (page 22). This ambiguity was reflected in the design of the results matrix with target indicators set after the closure of the operation in June 2013 and June 2014. As this was a one-off DPL, the results matrix and M&E arrangements should have been limited to this operation. Moreover, the wording of the first specific objective and its indicators could have been improved to target a strengthened legal framework for an energy efficiency strategy. Most indicators were designed to show implementation of structural reforms and achievement of outputs, instead of progress toward outcome.

This operation had a substantive results framework (Annex 2 of the PD), with adequate description of objectives, prior actions, medium-term (May 2012) and long-term (June 2013) actions, with assorted short-term (mid-2012) and medium term (June 2014) results indicators. However, this results matrix was unfit for a one-off operation to close on March 31, 2013. Association of GEF-funded and Bank-funded technical assistance to the DPL was a major opportunity for synergy in the pursuit of the loan's objective.

The major weakness in the design of this operation lied with the mismatch between the long-term reform agenda to step up energy efficiency and renewable energy, and the single-tranche and short term nature of the operation. The reform agenda deserved a series of programmatic DPLs, preferably coupled with investment and/or technical assistance projects. The mismatch was illustrated by the case of result indicators that were to be assessed well beyond the closing of the operation. It was unrealistic to achieve the expected results in energy efficiency and renewable energy in two years.


4. Achievement of Objectives (Efficacy) :


As the policy matrix did not make clear the distinction between the shorter-term slice of the program supported directly by the DPL and the medium and long-term actions/results of the Government’s program that were to occur outside the timeframe of the operation, efficacy toward the loan's objectives gave more credit to near term achievements and a little less to completion of medium and long-term reforms.

Develop a legal framework for energy efficiency strategy: Modest

Prior actions

Under this sub-objective, an Act on energy efficiency was promulgated as a prior action, thereby allowing the establishment of the White Certificates Program. Three medium-term actions were also completed as follows: (i) the annual review of the energy policy was adopted by the Council of Ministers (ii) the National Energy Efficiency Action Plan-2 (NEEAP-2) was submitted to the European Commission, and (iii) the Central Statistics office published a report on the implementation of energy efficiency measures, and (iv) the monitoring and evaluation of EE measures was upgraded. The status of long-term actions could not be assessed.

Outcomes

The performance indicator for this specific objective was achieved as a draft ordinance determining the level and method of calculation of the energy savings obligations imposed on energy suppliers for the implementation of the White Certificate Program was adopted. In summary, most policy actions under this sub-objective were achieved. Completed actions helped improve the legal framework and strategic thinking on energy efficiency, in addition to establishing monitoring schemes. However, the ICR reported that the overall legal framework for an energy efficiency strategy was not yet fully developed and implemented, as in particular the secondary legislation lagged behind the primary legislation, and the adequate governance structure has not been established yet.

Decrease supply-side energy use: Substantial

Prior actions

Two prior actions were completed as follows: (i) the Borrower provided incentives to increase the share of cogeneration by allowing cogenerators to price their bulk heat up to the average price of heat produced by heat-only boilers; (ii) the Borrower issued a draft regulatory statement to cover all the key areas of implementation of smart meters, and launched the implementation through the state-owned enterprise Energa. Two medium-term actions were completed as follows: (i) an Ordinance on the methodology related to Guarantee of Origin certificates (combined heat and power -CHP- and fuels used), including provisions for support of biogas from biomass and coal-bed methane (“purple” certificates), and (ii) the Government issued an ordinance to establish the “brown” certificate scheme to promote cogeneration from agricultural biomass. Other planned long-term actions with indicators set for June 2014 were not completed or could not be assessed.

Outcomes

Two performance indicators toward this sub-objective were also achieved: (i) adequate support for the implementation of the energy efficiency program was provided as measured by at least maintaining 2011 staffing and budget levels of the energy efficiency unit within the Ministry of Energy, and (ii) the Borrower allocated PLN 260 million in 2011 and leveraged PLN 1.7 billion for financing the thermo-modernization and renovation fund. While prior and medium-term actions, and progress indicators were achieved, results in the supply-side energy efficiency remained limited. The development of Ordinances to support the brown (agricultural biomass) and purple (biogas) certificates were adopted. In addition, the legislation supporting the yellow (gas) and red (coal) certificates was enacted by the President on April 7, 2014. However, the market continues to perceive the current energy environment as unstable.

Improve demand-side energy efficiency: Modest

Prior actions

Toward the achievement of this sub-objective, the Borrower implemented an Electronic Tolling System for heavy vehicles on major national road sections, as a prior action. Two medium-term actions were also satisfactorily accomplished: (i) a Smart Grids Law to be submitted to Parliament and (ii) a Draft Law on Energy Characteristics of Buildings were approved by the Council of Ministries, transposing the EU’s Recast Energy Performance of Buildings Directive. Other planned long-term actions with indicators set for June 2014 were not completed or could not be assessed.

Outcomes

Two performance indicators for this sub-objective were implemented as follows: (i) toward the installation of 200,000 smart meters, Energa has adjudicated the installation of 310,000 smart meters by the end of 2013, of which 109,000 were installed in 2012, and (ii) the second indicator was partially achieved as data on total fuel consumption and top down estimation of CO2 emissions was available, while complete data by category of heavy vehicles was not. Results towards the third objective were limited: (i) Full implementation of smart meters was behind target, despite the pilot installation of 109,000 smart meters. The benefits from the smart meters needed adequate infrastructure and incentive mechanisms as well as conducive legal framework, (ii) while the financing program was implemented, it was not coordinated with the National and Regional Environmental Funds and EU funds. On the positive side however, the financing for the Thermo Renovation fund was achieved and the level of co-financing exceeded, and the Fund is targeting the residential sector which is one of the most difficult sectors to reach for Energy Efficiency (EE) renovations; (iii) improvements in the building sector were slower than expected; new building standards have not been implemented, the enactment of revised energy characteristics of buildings has not been adopted, and registry for energy performance of buildings certificates, has not been introduced, and (iv) the transport component that focused on reducing emissions from heavy vehicles and their impact through an electronic toll collection system was partially achieved, while the Electronic tolling System (ETS) was successfully delivered and has been extended. The data is scattered among many institutions and is maintained in several sources, and the monitoring and evaluation of results needs to be improved to make this system more effective.

Renewable energy action plan: High

Prior actions

As a prior action, the borrower submitted its National Renewable Energy Action Plan to the European Commission, which details the country's commitment to a level of at least 15.0% of renewable energy use by 2020. The unique medium-term actions was not achieved as the draft of the Renewable Energy Law was not yet approved by the Council of Ministers. Other planned long-term actions with indicators set for June 2014 were not completed or could not be assessed.

Outcomes

The unique performance indicator which was achieved was the share of renewable energy in final energy increases exceeded the target (10.6% in 2010, against a target of 9%). In summary, the share of renewable energy in final energy has increased from 7.2% of final energy consumption in 2009 to 10.6% in 2011, and is projected to be 11% in 2013. Based on this trajectory, Poland is expected to meet its EU target of 15% of final energy consumption from renewables by 2020. However, increasing share of RE in the Polish energy system creates new problems. Renewable installations, once built, create operational problems as their marginal operating costs are nearly zero and unpredictable for solar and wind.

5. Efficiency (not applicable to DPLs):

6. Outcome:


Moderately satisfactory

Relevance of objectives was high, because Poland needed budget support to address treasury constrains, and a partner to help shepherd the energy reform in the country's quest to align with the EU energy efficiency standards and to increase the production of renewable energy. However, there were shortcomings in the statement of objectives in the loan's agreement, as most of them were stated as outputs. Relevance of design was modest, because of the mismatch between the long-term reform agenda to step up energy efficiency and renewable energy production, and the single-tranche and short term nature of the operation.

Efficacy was high for the fourth sub-objective as the share of renewable energy in final energy has increased from 7.2% of final energy consumption in 2009 to 10.6% in 2011, against a target of 9% and is projected to be 11% at end-2013.

Efficacy was substantial for the second sub-objective. Targets for performance indicators were achieved, although results in supply-side energy efficiency remained limited, and the market continued to perceive the current energy environment as unstable.

Results were modest for the first and third sub-objectives. Regarding the first sub-objective, while completed actions helped improve the legal framework, performance to establish a strengthened legal framework for energy efficiency was not yet fully developed and implemented, and the adequate governance structure has not been established yet. For the third sub-objective, (i) the benefits from the smart meters needed adequate infrastructure and incentive mechanisms as well as conducive legal framework, (ii) while the financing program was implemented, it was not coordinated with the National and Regional Environmental Funds and EU funds, (iii) improvements in the building sector were slower than expected as new building standards have not been implemented yet, and finally (iv) the impact on reducing emissions from heavy vehicles through an electronic toll collection system was partially achieved, and the monitoring and evaluation of results needs to be improved.

Because of the weak project design and that there were shortcomings in the project’s relevance and efficacy, overall outcome is rated Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:


Moderate

Across the political spectrum, there is a country’s commitment to achieving EU targets of reducing energy consumption and progress in establishing new renewable energy schemes. Bank's related GEF and technical assistance projects have been contributing to the likelihood of sustainability of the achievements of the DPL objectives. However, recent lower economic activity in Poland has reduced energy consumption, and resulted in lower energy prices. This creates an environment of lesser incentive to invest in energy efficiency. Energy efficiency and renewable energy policy reforms may be assigned a lower priority over the near term.

Concerns regarding competitiveness of Polish industry and the lobbying of different renewable energy technologies, as well as fossil fuel technologies, will continue to burden the processing capacity for reforms in the energy sector. Coordination among cabinet ministries, the energy regulator, regional and municipal Governments, as well as other public entities is still a challenge for the Polish administration, requiring continuous political buy in and champions to pursue the reform agenda.

The risk of social tensions and declining support for the Government’s Renewable Energy Program resulting from increases in energy prices and administrative costs for market participants to engage in energy efficiency investments is still relevant. Upgraded public communication and continued political support for energy efficiency are expected to support the continuation of the reform agenda.
On balance, the risk to development outcome is rated as moderate.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:

This operation was supported by a solid analytical work prepared by the Polish government, the Bank, the European Union, and the energy efficiency lessons learned from other European Union countries, which led to proper identification of objectives addressing the priorities of the country. This also followed the existence of a good dialogue between the Bank and the country's authorities and a Bank's strategy that was supportive to the Poland’s convergence towards EU living standards, and in the specific area of regional and global public goods.

However, there was a mismatch between the long-term reform agenda to step up energy efficiency and renewable energy, and the single-tranche and short term nature of the operation. The reform agenda deserved a series of programmatic DPLs, preferably coupled with investment and/or technical assistance projects.

Because of the above-mentioned mismatch, the results chain ended up to be inappropriate, as illustrated by the case of result indicators that were to be assessed well beyond the closing of the operation. The reform agenda deserved a series of programmatic DPLs, preferably coupled with investment and/or technical assistance projects.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

Supervision was provided through sustained dialogue and presence of Bank team members in the field. The Bank team maintained frequent dialogue with the client, including in conjunction with follow-up on GEF-funded and Bank-funded technical assistance. The presence of team members in the field permitted close monitoring of progress towards outcome indicators, and the collection of data and information from various line ministries in a timely manner.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government demonstrated commitment to policies promoted under the DPL, ensuring that adequate levels of human and budgetary resources were allocated and maintained to provide institutional capacity. All prior actions were completed on time and the Government is continuing to build capacity to implement the follow-up measures to ensure that the program remains on track with medium term actions.

The capacity of the energy team within the Ministry of Economy was stretched thinly as it has a fairly broad mandate and a modest team. Most Government bodies followed through on the program reforms and handled the challenge of preparing substantial amounts of new legislation. The Ministry of Finance and the regulator, the Unit for Renewable Energy (URE), were consistently responsive, helpful and supportive throughout the process.

Delays in the implementation of the primary legislation as well as supporting secondary legislation were symptoms of the limited capacity of the Ministry of Economy. There were some challenges
with timely implementation of the DPL triggers and actions as they were at times viewed as a disruption to normal work.

Government Performance Rating: Not Applicable

b. Implementing Agency Performance:

Implementing Agency Performance Rating: Not Applicable

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

This operation had a substantive results framework (Annex 2 of the PD), with adequate description of objectives, prior actions, medium-term (May 2012) and long-term (June 2013) actions, with assorted short-term (mid-2012) and medium term (June 2013) results indicators.

However, most indicators were designed to monitor implementation of structural reforms and achievement of outputs, instead of progress toward outcomes. The major weakness in the design of this operation lied with the mismatch between the long-term reform agenda to step up energy efficiency and renewable energy, and the single-tranche and short term nature of the operation. The mismatch was illustrated by the case of result indicators that were to be assessed well beyond the closing of the operation

b. M&E Implementation:

The presence of team members in the field permitted close monitoring of progress towards outcome indicators, and the collection of data and information from various line ministries in a timely manner.
There was a strong interlinked series of reforms, which reduced the risk of any move away from addressing the carefully defined program of reforms over the first year of implementation and also in the medium term.

Data was systematically measured and consolidated. Energy savings were calculated using a top-down method, published by the European Commission. Total final energy savings were calculated for the
national economy as a whole and by all the end use sectors. The method allows for demonstrating a direct relation between the implementation of energy efficiency measures supported by the energy policy. At the time of the ICR most of the indicators in the policy matrix have been updated since they were regularly monitored by the Government.

a. M&E Utilization:

The Central Statistical Office and the Polish National Energy Conservation Agency monitored energy savings during the DPL implementation period. The above two institutions helped to develop energy efficiency indicators for EU economies, and used them to analyze, identify and evaluate the results of energy efficiency policy. This exercise has helped the country to monitor and orient the course and progress of the reform agenda in energy efficiency and development of renewable energy.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

The program document provided an assessment of safeguards issues as follows: (i) the program supported by Poland Energy Efficiency and Renewable Energy DPL was not expected to result in negative environmental impacts, (ii) the proposed DPL was expected to accelerate energy efficiency and renewable energy interventions with a number of environmental benefits, and (iii) regulatory environment and institutional capacity in Poland was considered adequate to minimize and mitigate possible negative environmental impacts that could arise from scaling up wind power and the use of biomass energy. The ICR didn't provide an update on these issues.

b. Fiduciary Compliance:

There were no fiduciary issues raised either in the program document or in the ICR.

c. Unintended Impacts (positive or negative):

The ICR didn't report any unintended impacts

d. Other:

Not applicable



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

Two lessons stand out from the implementation of this DPL:

(i) The time-frame to implement a DPL operation matters, because it determines which kind of policy measures can be completed toward the achievement of identified outcomes. In the case of a single-tranche operation, the scope of outcomes should be scaled down, while a larger scope of outcomes can be envisioned in the case of a programmatic DPLs.

(ii) A strong analytical work and sustained policy dialogue with the country authorities have to underpin the design and approval of a DPL. Policy reforms require, among other things, a champion in the government as well as a strong constituency. A successful DPL has to gain adequate insights related to the above from preliminary political economy studies, and a sustained dialogue with the key reformers in the beneficiary country.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR provided a candid and complete picture of the DPL design and implementation, including detailed documentation on the country context. However, the assessment of the mismatch between the long-term nature of the reform agenda, and the single-tranche operation could have been done more adequately. While the ICR concurs with the fact that the one-off DPL operation was not appropriate for a wholesale and long-term sector reform agenda, it doesn't conclude that outcomes could not be achieved under the shortened loan time frame.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6DPL-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions