Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Energy Efficiency Program/industrial Sector


  
1. Project Data:   
ICR Review Date Posted:
06/27/2014   
Country:
Tunisia
PROJ ID:
P078131
Appraisal
Actual
Project Name:
Energy Efficiency Program/industrial Sector
Project Costs(US $M)
 31.8  36.1
L/C Number:
Loan/Credit (US $M)
 8.5  8.37
Sector Board:
Energy and Mining
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  11/04/2004
 
 
Closing Date
12/31/2009 11/30/2011
Sector(s):
Energy efficiency in power sector (65%), General industry and trade sector (29%), Banking (6%)
Theme(s):
Climate change (50% - P) Other financial and private sector development (50% - P)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Fernando Manibog
Ramachandra Jammi Christopher David Nelson IEGPS1

2. Project Objectives and Components:

a. Objectives:

The Project Appraisal Document (PAD, page 2) states that the Project Development Objective (PDO) is "to overcome barriers to the development of a sustainable market for energy efficiency products." The PAD defines the barriers as "institutional and capacity-related" and adds that "the project aims to establish energy service companies (ESCOs) as the main vehicle to guarantee a sustainable energy efficiency market."

The PAD (page 3) further states that the Global Environmental Objective (GEO) is "to achieve a deeper penetration of sustainable commercial energy-efficiency investment activities in Tunisia's industrial sector, by removing barriers and lowering transaction costs."

Schedule 2 of the Global Environment Facility Trust Fund Grant Agreement (dated December 13, 2004) states that "The objective of the Project is to facilitate the development of a sustainable market of energy efficiency Sub-projects through: (i) the removal of institutional and capacity related barriers; and (ii) the establishment of energy services companies.

Following IEG guidelines, this ICR Review assesses the project based on the objective stated in the Grant Agreement, while taking into account the important clarification in the PAD's GEO statement that the project specifically addresses the industrial sector.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

The project had three components:

Component 1 - Pilot Phase for Energy Efficiency (US$2.5 million at appraisal; US$ 2.07 million actual)
This component intended to establish and administer an incremental, output-based subsidy in order to enhance the development of energy efficiency projects in Tunisia's industrial sector. Prior to this GEF project, the Ministry of Industry (MOI), through its Competitiveness Enhancement Bureau (BMN), was already administering an Industry Competitiveness Fund (FODEC) that offered an energy efficiency subsidy of 13%. However, of the 1,202 applications to FODEC that were received from industrial companies by 2004, none included energy efficiency projects. Thus, using GEF funds with a view to increasing the total financial incentive to 23% , this component provided an additional 10% subsidy to be administered by Tunisia's Energy Efficiency Agency (ANME). To access this subsidy, as well as the separate subsidy of 70% for energy audits, the industrial applicant had to submit a Program Contract (CP) that included a 3-year action plan to implement an energy efficiency project -- on its own or with the assistance of an ESCO. If the latter is chosen, the company and the ESCO had to sign a turnkey Energy Performance Contract (CPE), in which the ESCO guaranteed the energy savings, and agreed to pay the difference if the full level of energy savings were not realized.

Component 2 - Partial Guarantee Fund (US$4.0 million at appraisal; US$4.4 million actual)
This component was aimed at promoting the use -- specifically by industrial companies -- of the "full-service ESCO model" in Tunisia, by providing ESCOs with access to guarantees. The ESCOs would pre-finance the entire energy efficiency project, including the investment costs, but would receive a 75% guarantee for their bank loans, up to a maximum of US$200,000. The Tunisian Guarantee Company (SOTUGAR) would administer the guarantee fund of US$4 million.

Component 3 - Technical Assistance(US$2.0 million at appraisal; US$1.9 million actual)
To encourage energy efficiency investments, this component supported awareness campaigns and training activities for industrial companies, as well as improvements in the capacity of technical centers, engineering firms, and other organizations that could potentially become ESCOs. The component also financed a Project Implementation Unit (PIU).

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

The GEF grant estimate was US$8.5 million at appraisal; the actual amount was US$8.37 million. At appraisal, the total project cost estimate was US$31.8 million, of which the Borrower would contribute US$4.9 million and local sources US$18.4 million. The ICR indicated that the actual total project cost was larger US$36.1 million, of which the Government of Tunisia provided US$5.8 million and private sources invested US$21.8 million to supplement the GEF grant. Both the Government and private sources contributed larger amounts than estimated at appraisal. The project closed on November 30, 2011, or almost two years beyond the original closing date of December 31, 2009.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Substantial

The project's objectives remain relevant to the latest Country Partnership Strategy covering the period of 2010-2013, by supporting two of its three pillars, namely: (i) employment, growth and competitiveness; and (ii) sustainable development and climate change. In practical terms, the project's relevance is evident from activities that (i) supported improvements in the competitiveness of participating industrial companies by reducing their energy bills, and (ii) promoted investments by local companies in energy efficiency. Through energy savings, the project's activities were also directly relevant to avoiding corresponding CO2 emissions. According to the Project Appraisal Document (PAD), the project's objectives was fully consistent at appraisal with the 2004 Country Assistance Strategy, which emphasized "the need to support the Government's efforts in the energy sector, with a focus on energy efficiency and renewable energies." (PAD, page 3)

Energy efficiency has also been and still remains a priority of the Government of Tunisia; this started in May 2001, when the Government adopted an aggressive strategy including twenty presidential decisions aimed at enhancing energy efficiency." (ICR, page 1)

b. Relevance of Design:

Modest

The project's design components, i.e., an operational subsidy, an investment guarantee, and technical assistance components, were straightforward but had important design flaws.

First, accessing the subsidy (Component 1) required the submission of a Program Contract between the industrial applicant and an ESCO -- but the ESCOs still had to be established, and the sequencing of creating the ESCOs and implementing the project components was too tight. The Guarantee Fund (Component 2) was also designed to provide a 75% guarantee to (yet non-existent) ESCOs for their Bank loans up to a maximum of US$200,000. The project design was over-optimistic in expecting ESCOs to make the jump from being non-existent to operating under the "full-service ESCO model", despite the immature energy efficiency market in Tunisia. The shortcomings in achieving this project design goal is discussed under Section 4 below.

Second, the reliance on non-existent ESCOs was further complicated by opting for two financial intermediation instruments, i.e., a subsidy window for participating industries and a guarantee window for ESCOs, to be overseen by different agencies, notably the Ministry of Industry and ANME. This made the project design confusing to potential stakeholders in Tunisia, where the idea of comprehensive energy efficiency investments in the industrial sector was new. Moreover, the project design did not establish formal and regular consultative mechanisms among the stakeholders.

With respect to the Results Framework, the selected output and outcome indicators were generally appropriate, although important ones on institutional capacity-building were weak or missing. For example, indicators related to stakeholder consensus were inadequate (see immediately above). The formulation of most of the indicators were specific, measurable, and time-based, i.e., they can be monitored. However, some of the causal links were weak. For example, a key link in the causal chain was absent, i.e., legislation to regulate the transfer of equipment ownership from ESCOs to industrial companies, yet this was not pursued during project preparation. Also, it does not seem possible to achieve the outcome of "a sustainable energy efficiency market for Tunisian industry" from having "At least 3 ESCOs are operational" as a modest output goal. Either the output goal had to be increased or the outcome goals scaled down, as they were mismatched and the baseline was low (competent engineering firms existed, but no ESCOs existed that could be scaled up) . The "number of projects approved" is also an inadequate indicator of a "sustainable energy efficiency market", as proven by the project's own experience of failing to mobilize adequate commercial financing despite the large number of projects approved. Another output goal was "At least 30 companies have ESCO-mediated projects" even though the ESCOs were non-existent. In fact, as early as project negotiations, the Government of Tunisia requested a deviation from the PAD's project design by requesting the Bank to make the guarantee facility also accessible to eligible participating companies themselves and commercial Banks (which were ineligible to use the guarantee facility in the PAD), because "these entities would be better placed in making the investments for the physical companies than the ESCOs, which at that point in time did not yet exist." (ICR, page 4). This again points to a hypothetical project design that excluded already existing, investment ready entities in favor of ESCOs that were yet to be created, with built-in risks for failure.


4. Achievement of Objectives (Efficacy) :

Objective: To facilitate the development of a sustainable market of energy efficiency.
The PAD specified that the market would be Tunisia's industrial sector, the financing vehicle would be energy efficiency sub-projects, and the specific instruments would be Energy Services Companies (ESCOs), a subsidy program, a Partial Guarantee Fund (PGF), and interventions to remove institutional and capacity related barriers.

Outputs:

(1) Establishment of a 10% subsidy program administered by the Energy Efficiency Agency (ANME) and a Partial Guarantee Fund (PGF). The ICR reports that "116 Program Contracts (CPs) linked to the project were approved and a total of 566 CPs were approved to benefit from the FNME subsidy, in large part because of support from the project." Analyses of the counter-factual and the degree of attribution to the project were not provided in the ICR.

(2) The PGF was fully committed under the project. However, future use of the guarantee funds were not agreed with the Bank prior to the project's closing date. Six loans benefiting from PGF guarantees were repaid by then, but there are no clear provisions for their future use.

(3) Number of projects reaching financial closure. Of the 116 projects approved, 81 have reached financial closure, representing a 65% partial achievement compared to the appraisal target of 125 projects. A total of US$26.9 million in energy efficiency investments were made over the project lifetime, including the 2-year extension.

(4) Number of operational ESCOs. The goal of having at least 3 ESCOs operational was only partially achieved. Although 10 ESCOs were licensed by ANME and 4 became operational, they were functioning more as auditing or engineering consulting firms that are paid a fixed fee rather than the full-scale ESCO model originally envisaged, i.e., funding the investments fully and getting paid from the energy saved. The industrial companies themselves had to mobilize the financing for their energy efficiency programs.

(5) Technical centers for monitoring and verification procedures related to energy efficiency investments. Two technical centers were formed via consultant contracts to monitor the energy efficiency investments: (i) technical center for construction materials, ceramics and glass; and (ii) technical center for mechanical and electric industries. A monitoring and verification procedure was developed before the project closed.


Outcomes:

(1) Energy savings were achieved among participating industries. Actual energy savings were 31,000 tons of oil equivalent (toe)/year, while in the future, expected energy savings based on the CPs were 51,000 toe/year, both of which exceed appraisal targets.

(2) Greenhouse gas emissions reductions were achieved, i.e., 710,333 tons for the 2005-2011 period including the 2-year closing date extension, compared to the original 636,422 tons projected at appraisal for the original project period.

(3) Improved planning and institutional capabilities led to the adoption of energy efficiency programs. A National Energy Efficiency Program was developed and a National Energy Management Fund was established.


Overall Assessment of the Achievement of the Project Objective:

Substantial

Targeted levels of program contracts, energy savings, and greenhouse gas reductions were partially, fully or more than achieved. However, the instruments--i.e., full-scale ESCOs and guarantees--that led to these achievements did not materialize or operate exactly as the PAD had (over-optimistically) designed originally. Nonetheless, what resulted from the project were modified versions that grew directly out of Tunisian realities, and were more suited to those market conditions.

With respect to ESCOs, the PAD's design that relied on ESCOs "as the main drivers for energy efficiency investments" (ICR, page 6) only partly materialized. Instead of full-scale ESCOs taking full risk by mobilizing investment funds and being paid out of realized energy savings, the participating industrial companies were the ones who paid for the physical investments, while the limited version of ESCOs that actually emerged were paid by those industrial companies on a contractual basis that included a large fixed fee and a small variable fee that is proportional to the level of energy savings achieved.

Regarding the Partial Guarantee Fund (PGF), two-thirds of the investments by the industrial companies were paid through credits guaranteed by the PGF. However, energy efficiency was new to the business portfolio of commercial banks and ESCOs were an entirely new concept. Thus, the commercial banks considered the guarantee only as supplementary insurance and still required traditional collateral from the industrial companies. In the end, the PGF acted as a double guarantee on top of the traditional collateral for loans by industrial companies.

Overall, a structure supporting a sustainable EE market emerged, although not quite in the form originally intended in terms of the type of ESCOs, and the easing of commercial bank requirements for financing EE projects.


5. Efficiency:

Substantial

Cost Effectiveness. According to the ICR's Annex 3-Economic and Financial Analysis (pages 31-32), the GEF subsidy had a greater leveraging effect than originally anticipated at appraisal, i.e., $13 (instead of $9) was actually mobilized from local sources including the Government as well as a greater participation by the local private sector. Greenhouse gas emissions reductions were achieved (see levels in Section 4 above) at an actual unit cost per ton (US$11.9) that is less than the appraisal estimate (US$13.4).

Administrative Efficiency. There were disbursement delays as stakeholders took the time in the project's early years to become familiar with the ESCO concept (see project design weaknesses in Section 3b above). There were also lags between ANME's Program Contract approval and the actual disbursement of the GEF subsidy, which was released only when the energy equipment was already in place. These required 3 extensions amounting to 2 additional years from the original Closing Date. Disbursement caught up and the entire GEF grant was utilized, following an action plan implemented by ANME in 2009 to help industrial companies to implement their energy efficiency programs and collect their respective subsidies more rapidly.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The relevance of project objectives is substantial as it remains relevant to the Country Partnership Strategy and the Tunisia's priorities in its energy and industrial sectors. The relevance of project design is modest given its dependence on the full-service ESCO model that did not yet exist, the complexity of its financial intermediation mechanisms, and weaknesses in key causal links in its Results Framework, particularly those related to building stakeholder consensus and critically necessary legislation that were not addressed during project preparation. The project's efficacy is rated substantial as targeted levels of program contracts, energy savings, and greenhouse gas reductions were partially, fully or more than achieved; the instruments leading to these achievements, however, did not materialize or operate as designed (over-optimistically) in the PAD. The project's efficiency is rated substantial in light of lower unit costs for greenhouse gas emissions reductions and higher resource leveraging effects, when comparing actual versus appraisal levels. The overall project outcome is moderately satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

It is uncertain whether the full-scale ESCO concept could succeed as the main vehicle for establishing a sustainable energy efficiency market in Tunisia. The existing ESCOs, while able to help in achieving energy savings, are really limited to providing consulting services because they are financially weak and will continue being unable to mobilize financing for large energy efficiency investments. A follow-up Bank project approved in 2009 that provides an energy efficiency line of credit has not shown much progress either, with less than 5% disbursed as of 2012. (ICR page 17)

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The Bank seriously underestimated the challenges of transferring the European and North American "full-scale ESCO model" into Tunisia, which had much less experience in energy efficiency investments, and where such projects were largely unknown among industries and banks, at the time of project preparation. That ESCO model was not the most appropriate one to attempt in Tunisia since, by the mid-2000s when the project was appraised, it has succeeded in scaling-up energy efficiency in only a few countries. Too little time was given between creating operational ESCOs along that full-scale model and implementing the project's physical and financial (i.e., subsidy and guarantee) components. Moreover, the Bank did not adequately take into account the prior importance of building personal contacts and trust for doing business in Tunisia. ESCOs could not have developed without entrenching this trust in a new business line first, yet the Bank's project design relied too heavily and too quickly on complex financial intermediation mechanisms. In short, "the Bank's team tested a theoretically appealing model, ESCOs, to a weak and underdeveloped market." (ICR, page 19) The project's goals were ambitious yet the necessary legislation (notably on the transfer of equipment ownership from ESCOs to the industrial companies) was absent or weak.

A conflict of interest also seems to have pervaded project preparation and design, since the project was based on a study done by the only ESCO operating in Tunisia at that time, i.e., the Societe Tunisienne de Gerance de l'Energie (STGE), which was created by the foreign firm Econoler in 1998. STGE went into bankruptcy soon thereafter because it was unable to develop a pipeline of energy efficiency projects, given the high transaction costs, the immaturity of the energy efficiency market in Tunisia, and mistrust of that foreign company by Tunisian industrial companies.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

The Bank supervision team adequately performed the regular supervision functions, including timely and balanced reporting including Aide Memoirs and Implementation Status Reports, carrying out a midterm review to identify measures for accelerating disbursements, and helping to develop an action plan to facilitate the application process for the industrial companies to obtain their subsidies. However, there were also some shortcomings. The Bank did not push strongly enough for the timely development and implementation of an Energy Efficiency Measurement and Verification Protocol, which is critically important to Performance Contracts (CPs) and the further development of the ESCO market. At project closing, there was also no clear agreement with the PIU on an exit strategy for the GEF Partial Guarantee Fund. Finally, the applicability of, and compliance with OP8.30 (on financial intermediation) should have been reviewed during the midterm review when the concept of an interest-free loan on investment costs was introduced.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government has been and continues to be a strong advocate for improving the efficiency of energy use in Tunisia. Prodded in 2005 by severe budget constraints and spikes in international oil prices, the Government developed an action plan to reduce energy intensity among its highest energy consumers in the industrial sector and established a specialized National Energy Management Fund (FNME) to provide grants for energy efficiency investments out of a tax on vehicle registration. The Government also re-assigned the management of the grant to ANME, given the inability of the then-existing FODEC fund under the Ministry of Industry to attract proposals for energy efficiency projects. However, the Government lacked legislation, and did not take adequate action, on transferring energy efficiency equipment from the ESCOs to the industrial companies upon completion of the CPs.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

ANME and the PIU played important roles in the achievement of project implementation targets. The PIU was proactive in conducting personalized campaigns to raise awareness about energy efficiency and the available subsidies among industrial companies. By the midterm review, the PIU was already directly facilitating the energy efficiency activities of 50 companies. When only 18% of the GEF subsidy (Component 1) had been disbursed with just one year left before the original 2009 closing date, the new PIU Director who assumed office in 2008, together with the Bank team, developed an action plan to improve disbursement during the remaining project implementation period. However, PIU was inadequately staffed during the initial two years of the project. The PIU was also slow in developing an Energy Efficiency Measurement and Verification Protocol, and did not have a comprehensive financial and market sustainability plan when the project closed. Finally, ANME could have been more straightforward and transparent in its administrative procedures for the approval and disbursement of subsidies.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

An actionable monitoring and evaluation (M&E) system was presented in the PAD (Annex 14, pages 76-78), with clear institutional responsibilities (including a dedicated consultant in the PIU at ANME), tracking and data collection functions, and reporting requirements. Since capacity-building indicators were weak, however, there were inadequate provisions for monitoring progress in key institutional aspects that were outside the direct physical and financial components. There were other weaknesses in the indicators: for example, "number of projects approved" was shown to be an inadequate indicator for the goal of establishing a sustainable market, when commercial financing proved difficult to mobilize. Another example was to set a target of 30 ESCO-mediated projects, when the ESCO were yet to be created and there was no prior ESCO experience whatsoever in Tunisia.

b. M&E Implementation:

During project implementation, the PIU at ANME used its own engineers instead of a consultant. The PIU conducted on-site visits to verify each energy efficiency investment prior to disbursing the subsidy claimed by the respective industrial companies.

a. M&E Utilization:

Physical and financial indicators were adequately monitored, leading to "a high-quality mid-term review report, which made several recommendations to improve project performance. Most of these were adopted by the PIU at ANME." (ICR, page 9) One important weakness of the M&E system was that the PIU did not use the International Measurement and Verification Protocol for verifying the energy savings. The ICR asserts, however, that the energy savings data "can be considered reliable because of the high technical expertise of ESCOs and PIU engineers." (ICR page 9) While high level of skills is probably an accurate assessment, it is unclear why the PIU did not use the Protocol despite prior agreement during project appraisal, as recorded in the PAD. (PAD, Annex 14) Instead, the PIU deferred plans "to develop the measurement protocol in future projects with the participation of key stakeholders." (ICR, page 9)

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

The project was rated a Category C and thus did not require an environmental and social assessment. During the implementation of the energy efficiency measures, the noise and dust emissions that were expected were minor and temporary.

b. Fiduciary Compliance:

The ICR provided assurances that "The financial management system in place was satisfactory and in accordance with Bank procedures." (ICR, page 10) The financial management action plan agreed in the 2004 Operations Manual was implemented satisfactorily by January 2005. The PIU produced financial monitoring reports in a timely manner and in line with the Bank's requirements. The PIU had one financial management specialist in its staff.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Significant
Significant
 
Bank Performance:
Moderately Unsatisfactory
Moderately Satisfactory
When there is a split rating, i.e., in this case, Moderately Unsatisfactory for Quality at Entry and Moderately Satisfactory for Quality of Supervision, the overall Bank Performance rating is aligned with the rating of Moderately Satisfactory for overall Outcome, per IEG Guidelines. 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The ICR (pages 21-23) provides clearly formulated and useful lessons based on the project's implementation experience. The most important ones are as follows:

1. Grants and guarantees are not sufficient to develop an energy efficiency market, whereas capacity building (awareness, coordination, and training) is critical. (This ICR Review would add that Government mandates alone are likewise insufficient if the necessary legislation and adequate trust in new financing mechanisms are not already present. Despite 2001 Presidential and 2004 Ministerial Orders mandating the development of an ESCO market, the full-service ESCO model did not materialize.)

2. The "full-service" ESCO model is not a viable instrument to leverage energy efficiency market development. The establishment of alternative ESCO models, which does not take investment risk, appears feasible as the case of Tunisia shows.

3. Subsidies that are administered on an output basis do not address the barrier of lack of "upfront capital" that is typical for energy efficiency investments. (This ICR Review would add that releasing the subsidy only upon completion of agreed outputs builds in a strong disbursement lag, particularly in innovative projects where awareness-raising is needed before energy investment activities can even occur.)

4. Keep project design simple. Having two financial intermediation instruments, i.e., a subsidy and a guarantee, led to different support mechanisms, eligibility criteria, governance structures, and complex administration requirements.

5. A guarantee fund is not sufficient to provide comfort to commercial banks and change the financing culture in a traditional system.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The quality of the evidence in the ICR is adequate, and the quality of analysis is strong. Its internal reasoning demonstrates a good understanding of the causal chains that underlie the project's results framework. The ICR is candid in its assessments of the project's shortcomings, and is generally well written, although it would have been useful to have a simple table showing the chronology of subsidies for energy efficiency in Tunisia. The lessons are well articulated and directly rooted in the ICR's evidence and analysis. The ICR is internally consistent and complies with OPCS guidelines, although it is somewhat long at 23 pages. Some repetitive material (e.g., on the level and origin of subsidies) could have been summarized in a well-designed table.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions