Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Support for Economic Expansion and Diversification (SEED)


  
1. Project Data:   
ICR Review Date Posted:
12/20/2012   
Country:
Zambia
PROJ ID:
P071407
Appraisal
Actual
Project Name:
Support for Economic Expansion and Diversification (SEED)
Project Costs(US $M)
 50.15  N/A
L/C Number:
C3966
Loan/Credit (US $M)
 28.15  28.86
Sector Board:
Cofinancing (US $M)
 22  N/A
Cofinanciers:
Board Approval Date
  07/24/2004
 
 
Closing Date
11/30/2009 11/30/2011
Sector(s):
Other industry (50%), Central government administration (15%), Mining and other extractive (15%), Agro-industry (15%), General agriculture fishing and forestry sector (5%)
Theme(s):
Rural markets (29% - P) Export development and competitiveness (29% - P) Infrastructure services for private sector development (14% - S) Biodiversity (14% - S) Other financial and private sector development (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Stefano Migliorisi
Michael R. Lav Navin Girishankar IEGPS2

2. Project Objectives and Components:

a. Objectives:


    Original Project Development Objectives (PDO)

    The original project objectives in the Project Appraisal Document were “to help reduce the vulnerability of the Zambian economy to shocks by supporting the diversification of its sources of growth” and to “improve management of Kafue and Mosi-o-Tunya National Parks to underpin the long term sustainable development of the tourism sector” (PAD, p. 3). In addition, the original Global Environment Objective (GEO) was “to improve the security of critical habitats and wildlife species in two national parks” (PAD p. 3).

    The objectives in the DCA (p. 24) were defined differently: "to support the Borrower's efforts to reduce the vulnerability of its economy to shocks by diversification of its sources of growth, including: (a) improving its sectoral policy and regulatory framework and strengthening the capacity of government agencies to implement them; (b) creating a conducive environment to encourage private sector investment; and (c) promoting and implementing conservation and management of biological diversity resources in the Protected Areas, including environmentally sustainable and socially inclusive livelihood of people living around targeted Protected Areas.” "Protected Areas" refer to Mosi-o-Tunya National Park, Kafue National Park, and Victoria Falls (DCA, p. 5).

    This ICR review uses the project objective cited in the DCA, It is more specific than the statement in the PAD and presents a more cogent framework of intermediate and final outcomes.

    Revised PDO

    The revised project objectives were to “improve the business environment for (i) sustainable tourism in the greater Livingstone area; and (ii) the gemstone sector.” (PP, p. iii). The amendments to the DCA presented the revised objective in the same way. The original Global Environment Objective (GEO) was retained.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 06/22/2007

c. Components:

Original Components

The project had four original components:

Component 1. Tourism and Protected Areas (original cost: US$17.34 million IDA plus US$4.00 million GEF; actual cost US$20.17 million IDA and US$3.97 million GEF). The Tourism subcomponent aimed at supporting three sets of activities: (i) policy, regulatory and institutional support for Ministry of Tourism, Environment and Natural Resources (MTENR) and its agencies; (ii) tourism investments and capacity building in the Livingstone area; and (iii) infrastructure for Livingstone and institutional support for City Council. The protected area subcomponent aimed at improving management, developing infrastructures and supporting biodiversity conservation in Mosi-oa-Tunya (MOT) and Kafue National Parks (KNP).

Component 2. Agribusiness Sector Development (original cost: US$2.50 million, actual cost US$0.10 million). The objective of this component was to increase the value added and competitiveness of the sector by accelerating the development of supply chain linkages from farm to market, improving quality and safety codes of practice, and encouraging new product development.

Component 3. Gemstones Sector Development (original cost: US$5.00 million, actual cost US$3.89 million). The purpose of the gemstones component was to support measures that promote gemstones production and trade and facilitate its inclusion into the formal economy, improve the sector’s socio-economic contribution at the regional and national levels, and encourage private sector investments.

Component 4. Implementation and Capacity Building (original cost: US$1.30 million, actual cost US$3.38). This component funded the salaries, equipment and capacity building for the Project Coordinating Unit and those agencies that play an active part in implementing the project.

Revised Components

During the 2007 restructuring, Component 2 was dropped, Component 1 was substantially redesigned, and Component 4 was changed into a Project Management Component. The three revised components were:

Component 1 - The Tourism Component (revised cost: US$18.66 million IDA plus US$4.00 million GEF; actual cost US$20.17 million IDA and US$3.97 million GEF) was simplified and incorporated three mutually reinforcing subcomponents:

    • Improving the national business environment for tourism through the development, communication and implementation of the new Tourism and Hospitality Act and related statutory instruments.
    • Improving the business environment for sustainable tourism in the Greater Livingstone Area: associated activities focused on supporting the rehabilitation of priority infrastructure for tourism in Livingstone, including the establishment of a One Stop Shop (OSS) for licensing tourism businesses, and the development and implementation of a Tourism Plan.
    • Facilitating development of sustainable tourism in Kafue National Park (KNP). This subcomponent continued to seek improvement of park infrastructure, management and biodiversity conservation to enhance the tourism potential of KNP and to attract private investment.
Component 2 - The Gemstone Component (revised cost: US$4.08 million, actual cost US$3.89 million) was retained, as originally designed.

Component 3 - The Implementation and Capacity Building Component(revised cost: US$2.90 million, actual cost US$3.38) was restructured to become a Project Management Unit Component.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Cost: Financial project costs at appraisal were US$50.15 million, when parallel funding is considered.

Financing: In addition to IDA (US$28.15 million) and GEF (US$4.00 million), the project received parallel funding from Norway (US$12 million at appraisal), and the Nordic Funds (US$6 million at appraisal).

Borrower Contribution: There was no borrower contribution.

Dates: There were significant deviations from planned dates during the course of the project. Approved on July 29, 2004, the project became effective on November 10, 2004, only 9 days behind schedule. However, the project had a slow start and after three years (end October 2007), it had disbursed only 41 percent of project funds. The project’s restructuring was approved by the Bank’s Board on June 22, 2007. The mid-term review (initially scheduled for February 15, 2007) was held over 2.5 years later (October 12, 2009). The project closing date was initially set for November 30, 2009. Such date was extended to November 30, 2011 during the 2007 restructuring and the project closed as per revised schedule.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Original Objectives

The original objectives were to help reduce the vulnerability of the Zambian economy to shocks by supporting the diversification of its sources of growth, and to improve management of Kafue and Mosi-o-Tunya National Parks to underpin the long term sustainable development of the tourism sector.

These objectives were:

    • consistent with the GoZ’s Sixth National Development Plan (SNDP) covering the period 2011-2015. The Plan's first overarching objective in fact aims in fact at accelerating infrastructure development, economic growth and diversification, while environment is a crosscutting issue. Specific sector strategies for gemstone and tourism are included in the SNDP.
    • aligned, although to a lesser extent, with the latest Bank’s Country Assistance Strategy covering the period FY08-FY11, particularly concerning the “providing access to currently inaccessible areas in selected national parks.”
The relevance of the original objectives is assessed as substantial.

Revised Objectives

The revised objectives were to improve the business environment for (i) sustainable tourism in the greater Livingstone area; and (ii) the gemstone sector. Although the agribusiness component was dropped, the project remained essentially focused on improving the infrastructure in national parks, and is therefore consistent with both the SNDP and the latest CAS.

The relevance of the revised objectives is also assessed as substantial.

b. Relevance of Design:

Original

The original design did not present a convincing causal chain between inputs, outputs and outcomes. The project aimed at reducing the vulnerability of the Zambian economy as a whole. Specifically, it was designed as a response to Zambia's copper crisis that was affecting an economy that remained dependent on exports of the mineral. Yet the project's resources (a little over US$30 million) were not commensurate to the task: it could support only a few pilots in tourism, agribusiness, and gemstones. Moreover, as noted in the Project Paper prepared for its restructuring (p. 3), the operation lacked a single locus of accountability for results.

The relevance of original design was therefore modest.

Revised

The revised results framework was less ambitious but not more convincing than the original one. It essentially collapsed outputs and outcomes. The revised outcomes (that is, improved business environment for tourism, secure critical habitats and wildlife species in two national parks, and improved business environment for the gemstone sector) were narrower and more readily traceable to project activities. They were also much more realistic in terms of inputs needed for a given output.

The relevance of revised design was therefore substantial.


4. Achievement of Objectives (Efficacy) :


Original PDO

The original PDO was to help reduce the vulnerability of the Zambian economy to shocks by supporting the diversification of its sources of growth (PDO1) and to improve management of Kafue and Mosi-o-Tunya National Parks to underpin the long term sustainable development of the tourism sector (PDO2). The actions supported by the project produced some important results concerning PDO2 and almost none for PDO1. However, the efficacy of the original PDO1 and PDO2 is rated as modest due to the weak causal link between these outputs and the ambitious outcomes, and the limited value of the indicators used to measure them.

Original Outputs

The project did not provide a clear set of outputs. It is however possible to reconstruct that it aimed at achieving farmer-agribusiness partnerships, the adoption of a new Tourism and Hospitality Act, the adoption of a business plan for the City of Livingstone Council, and a new National Mining Policy and regulatory framework. A Mines and Minerals Act was hurriedly passed in 2008, increasing rather than reducing the time needed to issue a license. The Tourism and Hospitality Act was ‘counterproductive and made the PDO more difficult to achieve’ (ICR p. 29). There is no information in the PAD concerning partnership of the business plan.

Original Outcomes

The original PDO was measured by several outcome indicators focusing on the following aspects: increased employment in agribusiness and income for the farmers involved; increase in the number of bed nights in the Livingston area and in the annual revenues of national parks covered by the project; increase in the population of key species; and finally increased production of amethyst and of revenue collection in the mining regions.The efficacy of the original PDOs is therefore rated as substantial for improvements in national parks, as the number of elephants and pukus increased (although two others, including buffaloes, declined), and revenues at the Parks almost quadrupled in nominal terms. The achievements of the original PDO in the areas of tourism, agribusiness and gemstones is rated as negligible since there was no visible outcome other than an increase in the number of bed nights.

Revised PDOs

The revised PDO was to improve the business environment for sustainable tourism in the greater Livingstone area; and the gemstone sector, while improving the security of critical habitats and wildlife species in two national parks. The protection of natural habitats and wildlife improved, while no significant improvement could be seen in the business environment for tourism or the gemstone sector. The efficacy of the revised PDOs is therefore rated as substantial on protection of natural habitats and wildlife and negligible on tourism and the gemstone sector.

Revised Outputs

The project achieved several of its revised outputs for the protection of critical habitats and wildlife species. The management effectiveness of KNP, as measured by the WB/WWF tracking tool, improved from 41 in 2004 to 63 in 2011. Management of the Park has been handed over to the Mukuni Management Trust. Almost all the staff at the park is properly lodged, and over 80 percent of the lodges and patrol areas are now accessible in all weather. In the gemstone sector, the National Mineral Policy has not been adopted. A Central Mining Cadastre was approved by Cabinet in November 2011 but could not have any effect as the project closed in the same month. Finally, as far as tourism is concerned, the legislation has not been passed, and the One-Stop Shop licensing facility in Livingston became operational only four months after the project closed. The only outputs achieved were the reduction of the number of annual operating licenses for tourism businesses from 5 to 1, and 18 km of roads rehabilitated against a target of 16.

Revised Outcomes

There has been an overall improvement in the protection of critical habitats and wildlife species with the help of more effective park infrastructure and skills (although only pukus increased by more than 5 percent, while elephants increased by less than 5 percent, and buffalos and red lechwe declined). Given the improved effectiveness of KNP’s staff, it is plausible to assume that the decline would have been more severe without the project. The number of days to get a tourism license increased by 50% rather than declining by a third while the time needed to obtain a mining license quadrupled for small scale mining, and almost quintupled for large scale mining -- an indication that the business environment for tourism and mining had not improved as expected. It is difficult to establish a counterfactual -- to establish whether the situation would have been worse without the operation -- since all related outputs were achieved at or after closing. The efficacy of the original PDO in tourism and gemstones is therefore rated as negligible, while the efficacy of original PDO for protection of habitats and wildlife is rated as substantial.


5. Efficiency:


The ICR analysis indicates and ERR of 32.4 percent, improved from the ERR estimated at appraisal mainly due to the restructuring and dropping of the agricultural component. In addition, the ICR justifies its moderately satisfactory rating for efficiency on the basis of "comparing actual results with project inputs." IEG does not share this view, in part because of the limited achievements of the gemstone and tourism objectives and the acknowledged cost overruns in various components. For instance, even under the relatively well-performing protection of habitat subcomponent, the ICR (p. 20) reports “relatively high operational unit costs for resource protection in the KNP, particular during the period when a bonus/incentive system was implemented (it should be noted that this bonus/incentive system was supported by Norway and not IDA/GEF).” Also, while the ICR argues that the US$0.5 million cost escalation of the PMU component was due to its elevated and more intensive role, it does not provide sufficient evidence that the restructured PMU provided the least cost alternative to meet the needs of the project.

The efficiency was modest both before and after restructuring.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
31.6%
100%
ICR estimate:
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Original PDO

The relevance of objectives was substantial. Yet, the modest relevance of design and modest efficacy of the original PDO were explained by the weak causal link between these outputs and its ambitious expected outcomes, as well as the limited achievements for most project activities. There were also shortcomings in efficiency, which was also rated as modest. Based on these ratings and the IEG/OPCS Harmonized Evaluation Criteria, IEG rates the overall outcome moderately unsatisfactory.

Revised PDO

After restructuring, the relevance of the project’s objectives and design were both substantial. However, there were also shortcomings in its efficiency and efficacy, both rated “modest”. The PDO of improving habitat and wildlife protection was at least partially achieved, but all other activities did not achieve their outcome Based on these ratings and the IEG/OPCS Harmonized Evaluation Criteria, IEG assesses the overall outcome as moderately unsatisfactory.

Overall Outcome Rating

As both the original and revised outcome ratings are both moderately unsatisfactory, IEG calculated an average outcome rating of 3 or moderately unsatisfactory.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:


IEG concurs with the ICR’s assessment of the risk to development outcome as significant, although for different reasons. There has been a lack of government funding for KNP, no Bank follow on operation and the unexpected withdrawal of Millennium Challenge Corporation (MCC) support to the KNP put the project’s outcomes at risk.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The project addressed relevant development challenges. However, it had no results framework (there was none in the PAD), and suffered from being conceived as a single, flagship instrument, to support and pilot approaches to diversification of the economy in the wake of the copper crisis. It sought to achieve high level outcomes that were beyond its reach and ended up with a complex set of components supporting a broad array of activities, spanning several institutions. It also lacked a clear accountability for results as the PCU had simply a coordinating and monitoring role. There was virtually no M&E framework and no provisions for an M&E specialist. The 2007 restructuring sought to address these weaknesses and improved the project design.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

The Bank spent substantial resources on supervision, with a total of 16 supervision missions and day-to-day interaction as both TTLs were internationally-recruited staff stationed in the field. The Bank deserves credit for being pro-active during supervision and identifying poor project performance quickly. Yet shortcomings were evident, including the following:

  • Performance problems were identified quickly, but the restructuring took almost two years to complete. Part of the problem was that guidance from Bank management was limited, even after ISRs flagged issues. The ICR (p. 27) noted that management “failed to better advise and direct the team on a number of issues that were raised for management attention in the project ISRs."
  • The Borrower, in its comments to the ICR, expressed concern about Bank TTLs micro-managing the operation (p. 50), particularly with regards to contract renewals for project staff.
  • Finally, the Borrower also expressed concern that the TTL who designed the project left three months after effectiveness. Nevertheless, there has been remarkable continuity in the TTL since then.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government's commitment was fairly limited, given that several outputs under its control were not achieved. For example, the two bills on tourism and mining were not passed, the number of days to receive a tourism or mining license increased rather than declined. Participation in Steering Committee meetings was also frequently poor. The Borrower itself (ICR, p. 50) noted that “Government did not own the process and was not fully invested in the activities undertaken. At times borrower requested activities would not move forward unless sitting allowances and/or out of pocket allowances were provided. Poor management and capacity on the part of the Borrower for certain subcomponent activities under their jurisdiction, resulted compounded delays and poor performance.”

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:

The Tourism National/Regional, Livingstone Infrastructure, Protected Areas and Gemstone components were implemented by the Ministry of Tourism, Environment and Natural Resources (MTENR), Livingstone City Council (LCC), Zambia Wildlife Authority (ZAWA) and the Ministry of Mines, Minerals and Energy (MME) respectively. The performance of ZAWA and LCC were moderately satisfactory, as they achieved most of their outputs. However, the performance of the two Ministries -- MME and MTENR -- was moderately unsatisfactory since both failed to achieve their regulatory objectives. The PCU performance was also moderately unsatisfactory till restructuring as the project lacked momentum and was poorly monitored. It did improve somewhat after restructuring, despite continued problems of high staff turnover.

Given that the Ministries’ weaknesses are already addressed in the Government rating (above), IEG rates the Implementing Agencies performance as moderately satisfactory.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The original project design did not include an integrated M&E management system and did not consider the need for an M&E staff in the PCU. The 2007 restructuring included a more coherent results framework and a clearer M&E system with indicators, monitoring tools and responsibilities.

b. M&E Implementation:

The project suffered in particular from the absence of a dedicated M&E officer. The SEED M&E plan and manual was designed only in 2008. A full time M&E officer with the required skills was hired only in 2009. As noted in the Mid Term Review, the M&E framework remained focused on expenditures and activities rather than results,

a. M&E Utilization:

Project data were not collected systematically and could therefore not be used in analyzing performance.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

The project was classified as environmental category B and triggered the following safeguard policies: OP 4.01, OP 4.04, OP. 4.09, OPN 11.03 and OP 4.12.The project complied with these safeguard policies.

b. Fiduciary Compliance:
Based on the available documentation reviewed by IEG, there were no fiduciary issues that arose during implementation.

c. Unintended Impacts (positive or negative):
None.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Unsatisfactory
Significant shortcomings in the project's efficacy (modest) and efficiency (modest) are noted in Sections 4 and 5. Also the relevance of original design was assessed as modest. Summary rating is based on methods prescribed by the IEG/OPCS Harmonized Evaluation Criteria. 
Risk to Development Outcome:
Significant
Significant
 
Bank Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The lessons IEG identified from the ICR and the overall experience of this project are as follows:
  • Projects relating to the reduction of licenses in important sector of the economy must be based on a thorough analysis of the political economy of existing regulations. Without a thorough understanding of the political dynamics of reform it is almost impossible to achieve significant results.
  • Complexity and excessive ambition should be easy to detect at project entry. There must be a stronger focus on avoiding costly mistakes by applying a more rigorous screening during preparation.
  • Monitoring and evaluation frameworks need to be comprehensive and put under the responsibility of competent staff whose recruitment should be an effectiveness condition.
  • In under-performing project, it may be difficult to know where pro-active supervision ends and micro-management begins. In such circumstances, the Bank should proceed with care, keeping in mind the need to ensure both political and operational ownership of the project by the Borrower.
  • Task team leaders need management support and guidance. It would helpful for the Bank to track delays in management responses.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR was candid and provided useful information to assess project performance. There were some issues of consistency, particularly in the assessment of efficiency and the justification for the outcome rating.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions