Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Integrating Environment And Forest Protection Into The Recovery And Future Development Of Aceh


  
1. Project Data:   
ICR Review Date Posted:
02/24/2014   
Country:
Indonesia
PROJ ID:
P098052
Appraisal
Actual
Project Name:
Integrating Environment And Forest Protection Into The Recovery And Future Development Of Aceh
Project Costs(US $M)
 17.53  17.62
L/C Number:
Loan/Credit (US $M)
   
Sector Board:
Environment
Cofinancing (US $M)
 17.53  17.62
Cofinanciers:
Board Approval Date
  02/10/2006
 
 
Closing Date
06/30/2010 12/30/2012
Sector(s):
Forestry (100%)
Theme(s):
Other environment and natural resources management (40% - P) Water resource management (40% - P) Other rural development (20% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
John Redwood
Ridley Nelson Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
According to the PAD, the project objectives were as follows:

1. to protect critical environmental resources and services from the Leuser and Ulu Masen forest ecosystems during the post-tsunami reconstruction process.
2. to contribute toward mitigating negative impacts of reconstruction on the forests of Aceh.
3. to contribute toward the mainstreaming of environmental concerns into planning processes.
4. to contribute toward building sustainable capacity and institutions for forest protection.

According to the two Grant Agreements, the overall project objective was: to ensure that environmental services provided by the Leuser and Ulu Masen forest ecosystems are effectively protected during the post-tsunami reconstruction process in Aceh. The discussion of the components in the Grant Agreements suggested three means to achieve this objective: (i) establishment of a multi-stakeholder governance framework; (ii) strengthening forest monitoring and protection systems; and (iii) development of appropriate institutions and capacity.

Following current IEG guidelines, the project will be assessed on the basis of the objective stated in the two Grant Agreements as the legally binding statement. The two were identical in this regard.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? No

Date of Board Approval:

c. Components:
1. Effective Protection and Sustainable Management of Leuser and Ulu Masen Forests (appraisal: US$ 9.64 million; actual: US$ 9,73 million)
a. Establish Multi-Stakeholder Governance Framework for Forest and Ecosystem Protection
b. Strengthen Forest Monitoring and Protection Systems and Procedures
c. Develop Sustainable Forest and Ecosystem Management Institutions

2. Integration of Environmental Concerns into Aceh's Reconstruction Development Planning Process (appraisal: US$ 4.63 million; actual: US$ 4.63 million)
a. Strengthen and Integrate Environmental and Conservation Considerations in Spatial Planning
b. Develop Environmental and Conservation Awareness and Education Efforts
c. Support Community-Based Forest and Ecosystem Rehabilitation for Livelihoods and Environmental Services

3. Effective and Transparent Project Implementation Support (appraisal: US$ 3.26 million; actual: US$ 3.26 million)
a. Strengthen Project Governance, Liaison and Reporting Processes
b. Support Internal Planning, Monitoring, and Evaluation.
c. Strengthen Financial Management, Procurement and Safeguards Oversight

Actual expenditures by subcomponent are not presented in the ICR but the increase in the reported actual cost of Component 1 presumably refers exclusively to subcomponent 1b. (see below).

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
This project was financed by two grants from the Multi-Donor Trust Fund (MDTF) for the reconstruction of Aceh and Nias established following the devastating earthquake and tsunami of December 2004: US$ 9.81 million from the Leuser International Foundation (LIF) and US$ 7,72 million from Flora and Fauna International (FFI). On November 16, 2012, the Bank approved a restructuring to make final adjustments to the allocation of proceeds among the disbursement categories, as well as additional financing of US$ 88,370 to increase the MDTF resources available to LIF in order to allow completion of aircraft inspection and repairs before the project closed. All other project activities were closed by June 30, 2011.

The project was formally restructured, with RVP approval, in October 2010 to: (a) incorporate a revised results framework for grants to both entities that implemented the project; and (b) reallocate more than 10% of the category allocations in the grant agreement with FFI. In practice, the Bank used the revised results indicators from the time of the third ISR dated November 2, 2007 and in August 2008 they were also reflected in the mid-term review in the form of a specific annex. However, the formal restructuring occurred more than three years later.

The closing date was extended by one year, to June 30, 2011 on June 14, 2010. The project was extended three additional times, to September 30, 2011, March 31, 2012, and June 30, 2012, respectively, mainly in order to complete procurement of three ultra-light aircraft for forest cover monitoring purposes.. Altogether, the project was restructured 7 times, according to the ICR (Table H, pg. iv). At the time the results framework was formally modified in October 2010 -- with the elimination of 9 of the outcome or output indicators with one retained and two modified -- US$ 17.13 million or 97.2 % of total project resources, had been disbursed.

The restructuring paper does not provide a detailed rationale as to why so many of the original performance indicators were dropped and replaced, noting only that "adjustments have been made to reflect changing realities...as well as a more realistic view of what the project can accomplish during its duration" (pg. 2). At the very least, this suggests that the project's original intended outcomes as given in the PAD were overly ambitious. Nor is it clear why the Region took so long to formally process restructuring of the Results Framework, although the task team subsequently informed IEG that this appears to have been due to a combination of other priorities, a change in lawyers in the Indonesia country office and processing delays with LIF and FFI.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
The project objectives as stated in the PAD were relevant at the time the project was appraised given both: (i) short-term concerns that earthquake and tsunami reconstruction activities could result in considerable plunder of local forest resources and an associated adverse impact on biodiversity; and (ii) longer-term ones with respect to continuing pressures on forest resources, including widespread illegal timber extraction, in the region. This broader concern continues to be part of the Bank Group's current (i.e., FY 2013-2015) Country Partnership Strategy (CPS) with Indonesia , one of whose four key pillars is "Pro-Green -- Ensuring Sustainable Development and Improving Disaster Resilience," including both biodiversity conservation and reducing deforestation and forest degradation by assisting in the implementation of the country's REDD+ strategy. However, the statement of the PDO in the Grant Agreements only partly reflected actual project content, predominantly the shorter-term aims with less focus on sustainability. Therefore the relevance of objectives is rated Modest.

b. Relevance of Design:
The PAD contained a "Project Results Summary" whose objectives generally correspond to those in the Grant Agreements i.e., the overall PDO is "environmental services from Leuser and Ulu Masen forest ecosystems are protected during the post-tsunami reconstruction." Those for the three Components were: effective protection of sustainable management of Leuser and Ulu Masen forests, integration of environmental concerns into Aceh's reconstruction and development, and, effective and transparent project implementation support. The project design, therefore, appropriately sought to combine both immediate short-term protection of these threatened forest and biodiversity resources and the building of capacity for their medium and longer-term sustainable management and protection.

However, as observed above, the longer-term objectives of the PAD were not reflected in the objective statement of the two Grant Agreements. Thus, there is a "disconnect" between the PDO, as stated in the Grant Agreements, and the actual component content of the project and the PAD statement of the objectives. In addition, the Results Framework, at least in terms of most of the original outcome and output indicators quickly proved to be inadequate, and needed to be substantially reformulated early on in project implementation, even though they were not formally changed until several years later.

As the ICR itself points out, there were other shortcomings in project design, both in having been "over-optimistic in planning to complete the many project activities in a five-year period" (para. 23, pg. 7) and in relation to project implementation arrangements, more specifically: (a) incomplete involvement of some pertinent government agencies, especially the local forest service agency; (b) failure to include local law enforcement agencies; and (c) an "ineffective" PCU resulting in "inadequate coordination, a reduced sense of ownership by stakeholders, and less than optimum outreach and dissemination of project information and materials." Finally, some items included in project design, such as plans for establishment of two Multi-stakeholder Councils and almost all of the construction works (i.e, for airplane hangers and airstrips and for a Conservation Center and other buildings), were eventually dropped, although this was due in part to implementation problems rather than to design flaws per se..

On balance, however, the relevance of design is rated Modest.


4. Achievement of Objectives (Efficacy) :

The overall objective as stated in Grant Agreements was: to ensure that environmental services provided by the Leuser and Ulu Masen forest ecosystems are effectively protected during the post-tsunami reconstruction process in Aceh. The PDO, as stated in the Grant Agreements, was clearly timebound in that it refers specifically to the period during which Aceh was undergoing reconstruction even though the project also contained elements that were concerned with the longer-term sustainability of forest protection. These were explicitly reflected in the objectives as stated in the PAD. Project efficacy will be assessed both in terms of key outputs and overall outcome with the latter referring specifically to the PDO as stated in the Grant Agreements. However, the achievement of the longer term activities are also discussed.

Outputs:

As noted in Section 2a above, the project sought to achieve the objective by three means: (a) establishing a multi-stakeholder governance framework for forest and ecosystem protection; (b) strengthening forest monitoring, management protection systems and procedures; and, (c) development of appropriate institutions and capacity for forest and ecosystem management and protection.

Multi-stakeholder management arrangements: Five such arrangements, exceeding the target of two frameworks, were established and adopted by October 2012, although the ICR does not explain what these "frameworks" consisted of or whether they corresponded in part to what the PAD's Project Results Summary described as a "multi-stakeholder institution for management of Ulu Masen ecosystem legally established by end of project." Forest management protection systems also seem to have been strengthened, at least in the short run, since, according to the ICR (para. 56, pg. 15), the project "exceeded expectations in terms of engaging communities and building capacity to monitor and enforce issues related to illegal logging, poaching and forest encroachment," while "the community ranger program and Conservation Response Units were particularly successful in achieving conservation aims, community engagement and livelihoods support". This also helped to reduce conflicts between communities and wildlife, especially elephants.

Forest monitoring, management and protection system: The three-tiered monitoring system was only fully established after multiple extensions of the project closing date because of the delays in the procurement of light aircraft. However, interpretation of satellite imagery and monitoring of deforestation by local communities were effectively supported under the project, and rented aircraft were used until the others could be acquired. The ICR does not present evidence to affirm that any of the four outcome indicators originally established for this activity in the PAD (multi-stakeholder management institution for Ulu Masen, legal record of strengthened conservation status of two ecosystems, government institutions responsible for forest protection collaborate through formal established mechanisms, and finance generation initiatives meet at least 30% of conservation management needs) were, in fact, met. In the absence of such evidence, IEG assumes they were not. In fact, the ICR (para. 61, pg. 17) acknowledges that the project did not set up any formal regional stakeholder bodies/consultation structures, which was one of the revised performance indicators introduced in October 2010 i.e to support and strengthen 3 existing or newly created multi-stakeholder bodies/consultation structures for forest/conservation management. However, all other revised indicators for this component but one were fully achieved or exceeded. The one not met was the target that aimed at 95% of the initial 1,000 forest guards contracted by Dinas Kehutanan to have received training and equipment support by end of project. Only 65% was achieved.

Development of appropriate institutions and capacity for forest and ecosystem management and protection: The ICR (para. 60, pg, 16) observes that the outputs in relation to this objective were "generally satisfactory and will be largely continued as outlined in the sustainability plans of LIF and FFI." However, it also observes (para. 61, pp. 16-17) that "it is as yet too soon to see the impact of these training and support activities on performance of the various agencies." It is of concern that, as the ICR acknowledges, since the two NGOs managed the project, "government agencies -- in particular Local Forest Service Agency (Dinas Perkebunan dan Kehutanan), Provincial Forest Police (POLHUT), and Provincial Police Service (POLDAT) -- had no executive role or budget. Therefore, involvement in project activities depended more on personal interest than any agency directive." The ICR also reports that "such relationship (sic) were often more successful in achieving collaboration between stakeholders than trying to set up a new mechanism, yet were often highly dependent on individuals and thus are difficult to sustain over time."

Based on the evidence presented in the ICR, while the project appears to have led, together with other non-project factors, to the effective short-run protection of these forests, its contributions to its implicit longer-term sustainable management and institutional capacity building are less evident.

Longer-term Capacity.

The KPIs went further than the Grant Agreement specification of the objectives in establishing a longer-term capacity intent in two main areas: (a) integration of environmental concerns in Aceh's reconstruction and development; and, (b) provision of technical advisory services and material assistance to support and maintain governance, liaison and reporting processes. The ICR presents insufficient information to make an informed judgment with respect to the achievement of some of the original and revised KPIs, However, reportedly the main reason why the original results indicators were changed was that, relatively early in project implementation, the project team realized that they were too ambitious and difficult to effectively measure.

Integration of environmental concerns in Aceh's reconstruction and development by: (a) implementing measures to integrate environmental and conservation considerations into spatial planning; (b) development of environmental and conservation education and awareness programs; and (c) provision of support, including through sub-grants, for community development initiatives for forest and ecosystem rehabilitation for livelihoods and environmental services.

The project did integrate environmental concerns into Aceh's spatial planning at both the regional and district levels, exceeding the target by 100% (2 instead of 1) with respect to the former and meeting it exactly with respect to the latter, at least in the form of draft planning reports delivered to local and provincial governments. According to the ICR, a spatial plan for Aceh had been accepted by the Ministry of Forestry and was "pending adoption" by the provincial Parliament, while five district plans had been adopted and two others were "pending formal adoption". It also affirms that, "if implemented," these plans "will reduce land use and resource conflicts and negative environmental impacts on forest-dependent wildlife and communities near or downstream to the two ecosystems," thereby providing "a major benefit for future conservation and management" of these ecosystems. However, as noted elsewhere in the ICR (para. 27, pg. 9), "at grant closure the forest spatial plan still had to be ratified and there is as yet no clear indication of the strategy to be adopted for conservation and development in Aceh." Sub-grants totalling nearly US$ 330,000 were provided to NGOs. This far exceeded the initial target of US$ 40,000, presumably -- although this is not entirely clear in the ICR -- to support alternative livelihood activities with the objective of reducing community dependence on forest resources.

Environmental education programs were also delivered through schools and over 1,000 teachers were trained, doubling the initial target. Thus, the revised targets were essentially met or exceeded. However, all four of the initial output indicators were dropped when the project was restructured suggesting that none were effectively met.

Provision of technical advisory services and material assistance to support and maintain governance, liaison and reporting processes in project management. This was to include management of internal planning and communications, implementation and policy compliance; and to ensure compliance with financial management requirements, procurement procedures and safeguards policies. Here too most of the output indicators were fully met, and, in this case, only one of the original indicators -- annual meetings of Steering Committee carried out -- was dropped, although two of the other three were slightly modified.

Outcomes:

The project does appear to have contributed to the protection of Aceh's forests from large-scale degradation and encroachment between 2006 and 2012 in spite of heavy timber demand for reconstruction after the 2004 tsunami and the return of ex-combatants to their villages and farms near the forest following the 2005 Peace Accord. As a result, 96.9% of the 2006 forest cover, or more than 2.6 million hectares, remained in 2010, the most recent date for which data were available according to the ICR (revised PDO Indicator 1, pg. iii), This exceeded the end project target of 95%.

According to the ICR (para. 54, pg. 15), the project achieved both its short-term objective of protecting Aceh's forest ecosystems during the reconstruction period and its medium-term objectives -- which were not explicitly stated as such in the Grant Agreements, but are implicit in Component 2 and some of the KPIs i.e. mainstreaming environmental concerns into the planning process and building sustainable capacity and institutions for forest protection. The ICR provides ample evidence to support the first of these claims, including with respect to the project's revised key performance indicators (KPIs), but less information to sustain the latter one, particularly in terms of building sustainable capacity and institutions for forest protection.

However, there are questions about project attribution. The ICR observes (para. 57, pg. 16) that "part of the success in protecting Aceh's forests was due to the efforts of other donors to source wood from outside of Aceh". Also contributing was the former Governor's 15 year moratorium on logging imposed in 2007 "which facilitated the detection of illegally sourced timber," and which "together with other conservationist policies...and Aceh green initiatives ... changed attitudes toward forest conservation and demonstrated GOA's commitment to protecting the province's forest resources." As a result, while this "political environment facilitated" project implementation, as the ICR (para. 57, pg. 16) correctly points out, it also makes it difficult to assess how much is due to project actions and how much to actions outside the project.

In IEG's meeting with the project team, in discussing the question of attribution, it was pointed out that, project and non-project interventions reinforced one another such that effective enforcement of the logging moratorium depended in good measure on the monitoring information provided by the local communities directly supported by the project. In this connection, it is noteworthy that over 80 law enforcement operations were carried out that "resulted in the arrest of 145 people and the confiscation of illegal timber, vehicles and chainsaws, as well as the closure of three illegal sawmills" (ICR para. 40, pp. 11-12) In addition, both the provincial government's strong support for the project's objectives and activities and the parallel forest conservation measures mentioned above were a reflection of the newly elected Governor's strong personal commitment to forest and biodiversity protection in Aceh. Fortuitously, the new Governor was an environmentalist himself and had been a former staff member of one of the two NGO grant recipients and co-implementing agencies, FFI.

On balance, based on the above analysis of the project outcome and attribution, project efficacy in relation to the overall objective in the Grant Agreements is rated Substantial.

5. Efficiency:

No economic or financial analysis was undertaken at appraisal or for the ICR. Nor is any cost-effectiveness analysis contained in the ICR, which limits itself to observations concerning the benefits from conservation of Aceh's forests in relation to those from deforestation based on other studies and a statement regarding the appropriateness of selecting two experienced specialized international NGOs to implement the project "given the significant disruption of Government services" following the tsunami. However, it is relevant to point out that, despite the two and a half year extension of the project closing date, there was only a very slight (0.5%) increase in total project costs in relation to the appraisal estimates. In addition, a study published by Aceh Green in 2010 concluded that conservation of Aceh's forests was expected to yield higher economic benefits (US$ 13,4 billion) than deforestation (US$ 12 billion) over a 30 year analysis period (ICR, para. 62, pg. 17).

Information subsequently provided to IEG by the project team, however, suggests that project resources were used efficiently, especially in relation to personnel costs, as only local staff and consultants were involved in addition to the arrangements with local communities to carry out forest monitoring activities. In addition, a number of the project targets were significantly exceeded, as detailed in Annex 2 (Outputs by Component) of the ICR, such as the number of conservation plans prepared and delivered to local governments (9 versus 6), the number of school teachers trained (1007 vs 500), the number of ecoclub members recruited (9,760 versus a target of 1,200), and the amount of small grants provided to local communities (US$ 328,500 versus the initial target of US$ 40,000, due to a reallocation of resources to this subcomponent), without any overall increase in project resources, suggesting an effective and flexible use of project resources. Project resources were also used to leverage -- and, thus, were directly complemented by -- resources from other sources for community livelihood support activities. As a result, efficiency is rated, on balance, Substantial.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Based on the analysis above, the relevance of project objectives and design are both rated Modest, due in part to the only partial reflection of project intent, as stated in the PAD, in the corresponding Grant Agreements and the inadequate Results Framework. Efficacy is rated Substantial largely due to the achievement of forest protection. Efficiency is rated, on balance, Substantial. The overall project outcome rating is Moderately Satisfactory. The project achieved its timebound development objective as stated in the Grant Agreements and met or exceeded most of its revised performance indicators with an efficient use of resources. However, it is unclear whether this was primarily due to the actions of the project itself or to exogenous factors, particularly a logging ban that was implemented in parallel to it. In addition, the project evidenced several significant design shortcomings that were not formally resolved during implementation in a timely manner or, in the case of the PCU, at all.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

According to the ICR (para. 27, pg. 9), at the time the Grants closed, the provincial spatial plan had not been ratified and "there is as yet no clear indication of the strategy to be adopted for conservation and development in Aceh in the future. The efficiency of mitigation measures is therefore uncertain in the longer term," clearly suggesting that the risks to the development outcome rating may, in fact, be more serious than the ICR concludes on the grounds that the two implementing agencies had "prepared detailed and comprehensive Sustainability Plans and have already secured funding for their implementation" (para. 71, pg. 19) The fact that such NGO plans exist and are funded does not necessarily "ensure that key AFEP [i.e, project] activities will be taken up, managed and followed through by the appropriate agencies and bodies," which are not themselves specifically identified by the ICR. In addition, the ICR also correctly points out that "protecting the forests and their resources and services will require continuing vigilance and support, and the integrity of protected forest areas cannot be taken for granted."
Of more concern is the statement that "there are signs that the provincial administration elected in April 2012 may be less committed to forest conservation goals than the previous government" and that "after the project closure, there could be efforts to cancel or shorten the 15-year logging moratorium, grant new logging concessions, weaken Provincial Spatial Plans, or reduce the size or change the status of some protected areas". Overall, the Moderate rating proposed by the ICR is not consistent with the text, which ends by stating "although the planned project outputs did not include the moratorium or final spatial plans, and these threats may not materialize, sustainability of the overall project rating could be at higher risk, despite mitigating factors such as greater public awareness, engagement of NGOs and CSOs, and empowerment and participation of local communities in forest conservation" (para. 72, pp. 19-20). Subsequent to the ICR, the project team informed IEG that the issues raised in the ICR may have been overstated and that future prospects may be more positive than the ICR suggests.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:
Given the emergency situation, the fact that project design later proved to have involved some notable shortcomings, particularly with respect to the failure to set up an effective overall PCU is perhaps understandable. However, the project's Results Framework was later found to contain overly ambitious and unmeasurable outcome indicators that needed to be replaced through a significant restructuring that took place in October 2010. The ICR observes that "initially the project concept was on environmental protection and stronger policing, which was modified to give greater emphasis to community involvement in conservation and environmental education," but that, "with hindsight, other innovative ways of increasing resources for policing could have been considered, but a project financed through grants to NGOs would not be an effective mechanism for this sort of support." This notwithstanding, project design could have included ways of providing capacity building support to pertinent public sector agencies as well, which would be important in terms of the project's longer-term sustainability objectives.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
The ICR correctly indicates that, despite regular missions and supportive Bank management, supervision failed to correct the initial design shortcoming in relation to the ineffective overall project PCU and "achieving cohesion" between the two implementing agencies, "thus allowing the problem to become more serious." Continuity of Bank staffing was reportedly also poor since 2009 due to frequent changes in TTLs, in the ICRs words, "leading to delays in revising allocations across budget categories for both FFI and LIF and thus disbursement, as well as processing of restructuring," which, in fact, occurred very late in the day in relation to overall project disbursements.

In this regard, it is unclear why the Bank took so long after the mid-term review (held in August 2008) to formally restructure the project's Results Framework, which did not occur until December 6, 2010 -- or nearly two and a half years later, even though, according to the restructuring paper dated October 30, 2010, this "slight modification" of the logical framework "was suggested in the [project] mid-term review". In this regard also, while ISR # 3 of November 2007 identified as a pending action that the project lawyer should revise the grant agreements to incorporate the revised Results Framework, this appears not to have happened. This would appear to be a lapse on the Bank's part. In addition, there is a partial "disconnect" between the statement of the project objective in the Grant Agreements with its focus on the short run (i.e., the Aceh reconstruction period) and the project's apparent actual aim at medium and longer-term sustainable management concerns. This is a case where the PDO, as well as the Results Framework, should have been formally revised in the Grant Agreements.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
Overall provincial government support of project objectives and activities appears to have been positive, although the ICR (para. 81., pg. 21) reports some "instances of relaxation of GOA's vigilance -- as for example in the case of the Tripa oil palm plantation" (details not provided). The provincial Governor who was elected after the project had started appears to have been strongly committed to the project's objectives and took complementary actions to support forest protection, which may have been as important, if not more so, than the project itself in achieving the positive outcome in this regard. However, the failure to establish "an effective and functioning PCU" was identified by the ICR as a significant "shortcoming." This notwithstanding, considering the very difficult post-disaster challenges face by the provincial government at the time this project was implemented, on balance, its performance was largely positive.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
Although facing a number of constraints, the two implementing NGOs -- LIF and FFI -- generally appear to have carried out the project activities for which they were responsible in an effective way. One of the two (FFI) was considerably more proactive than the other however. The ICR (para. 85, pg. 22) observes that "frequent changes in team leaders and staff at the IAs and Bank as well as early difficulties in following financial procedures specified by the Bank MDTF [Multi-donor Trust Fund] delayed implementation." On the other hand, as a QAG review pointed out, "the two IAs were not helped by the situation of having multiple donors" which added to the reporting and other administrative requirements.

However, the two NGOs were not the only planned implementing agencies. The ICR omits the PCU in discussing implementing agencies. The lack of an effective PCU was also an impediment to implementation.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
M&E design was poor. The ICR attributes this in part to the fact that "the project was designed and implemented quickly in response to what was considered an emergency situation." The Key Project Indicators (KPIs) in the Results Framework were formally revised in October 2010 (after more than 97% of project resources had been disbursed) to "make them more logical, measurable and reflective of the project objectives based on recommendations of both the MDTF and project mid-term review" that occurred in August 2008. The main emphasis of the original M&E framework was environmental monitoring, particularly that of forest cover, but there was no provision for surveys on project impacts on affected communities, nor was there clear baseline data or a system to monitor the detailed project indicators. In addition, the two implementing agencies (IAs) lacked capacity to collect, analyze and report on such indicators, partly due to the failure to set up a "functional" Project Coordination Unit (PCU) and "difficulties in field-level processing of performance indicators," according to the ICR.

b. M&E Implementation:
The individual IAs adequately collected and reported M&E information related to physical implementation progress and the "degree of success in conserving forest area." However, overall project monitoring information was "constrained" due to the "weak state of project coordination and collaboration" between LIF and FFI. Apparently, surveys were later undertaken to provide baseline data for project indicators as well as for biodiversity and wildlife. The ICR concludes that "data quality and timeliness would have been better had there been a fully functional PCU to oversee and collate information" from the two IAs. In addition, the extensive delay in procuring the light aircraft meant that they could not be used for aerial monitoring of forest cover as proposed at appraisal. As a result, project proposals for construction of hangers and landing strips were dropped, and commercial aviation services had to be contracted in order to monitor hot spots where deforestation activity had been detected on the basis of satellite imagery. Ground-based monitoring, on the other hand, reportedly "exceeded expectations" in terms of community engagement and capacity building.

a. M&E Utilization:
M&E data was used to produce information on the project for a "broad audience," reportedly helping to enhance public support for it together with increased awareness of forest conservation priorities, and also to "strengthen project outcomes," for example, the reporting of a large number of "forest offenses" in Ulu Masen to the police, leading to the aforementioned "arrest of 145 people and the confiscation of illegal timber, vehicles and chain saws" and the closure of 3 illegal sawmills. Satellite monitoring also provided a measure of the project's success in preventing forest loss. On the other hand, the ICR (para. 38, pg. 11) also affirms that "given the shift in strategic focus towards strengthened livelihood support, an evaluation of the project impact on living standards of communities would have been helpful."

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
At appraisal, the project triggered safeguard policies on environmental assessment, natural habitats, forests, involuntary resettlement, indigenous peoples and cultural property, but was classified in Category B. The ICR affirms that, in practice, the potential issues identified at appraisal in relation to the latter three policies "did not materialize" and that the project was rated "Satisfactory" in safeguards performance throughout implementation. On the other hand, "one case was reported in the Leuser ecosystem of two villages relocated due to flooding," which were "organized by the villages themselves." While these relocations were reportedly not related to the project per se, it provided technical assistance on new sites and rehabilitation of the former ones.

b. Fiduciary Compliance:
Financial management and procurement performance, while reported to have been generally "adequate" by the ICR, in practice appear to have been mixed. Audits of the two IAs were unqualified, but there were a number of FM issues early on due to the lack of IA staff familiarity with Bank procedures and internal control issues and shortage of trained finance officers, while "outstanding advances and some FM weaknesses were only resolved by LIF late in the project." Similarly, in the case of procurement, while dedicated specialists were assigned "for the most part" and adequate control systems were reportedly in place, the importation of ultralight aircraft through LIF, for forest monitoring purposes, was delayed considerably due to lack of clarity regarding their possible exemption from import duties, resulting in the last three project extensions. In addition, timely submission of quarterly and annual reports "remained an issue" throughout implementation.

c. Unintended Impacts (positive or negative):
No unintended impacts were reported by the ICR.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
The project's main objective was achieved, but this may have been due as much to exogenous factors, including a logging ban, than to the project itself. In addition, the operation was characterized by several key design shortcomings. These included a disconnect between the PDO as stated in the Grant Agreements and the actual content of the project and an original set of performance indicators that were both overly ambitious and nearly impossible to measure, thus needing to be largely replaced. 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Moderately Satisfactory
Moderately Unsatisfactory
There were significant shortcomings in both quality at entry and supervision including a nearly two and a half year lapse between the time of the mid-term review and the formal restructuring of the Results Framework, while the PDO was not reformulated in the Grant Agreements to better reflect actual project content (i.e., medium and longer-term, as opposed to just short-term objectives. 
Borrower Performance:
Satisfactory
Moderately Satisfactory
Despite a generally positive performance under very difficult circumstances, an effective PCU was lacking, there were some early shortcomings and some imbalance in implementing agency performance including the PCU. 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The ICR presents several useful lessons as follows (the first one slightly modified by IEG). IEG adds a fifth:

1. In environment-focused projects with a forest protection element, even where there is effective community protection, Government action is essential, especially for law enforcement and prosecution.

2. Demonstrating how conservation and protection activities directly benefit communities is more effective than rhetoric alone.

3. Conservation work, especially in a post-disaster context, needs to take the need for jobs and livelihoods into account, both for economic and local political reasons.

4. An environmental project, such as the present one, can effectively leverage resources for its stakeholder communities and vulnerable groups such as the poor and women.

5. The statement of the PDO in the legal agreements should fully reflect actual project objectives and content.

A more specific lesson from this particular case is that, given the longer-term effective protection and sustainable management needs of the two forests in question, greater attention could have been given to: (i) supporting and strengthening the pertinent public sector policies and institutions with responsibilities over these issues; and to, (ii) measures to enhance the longer-term sustainable protection and management of these forests since they will continue to be threatened, especially once the current logging moratorium is lifted.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is well-written and forthcoming, and also draws useful lessons, but there appears to be an inconsistency between the text and some of the ratings. Another weakness is the lack of useful information regarding project efficiency in the use of resources -- i.e., cost-effectiveness, although the last TTL was subsequently able to provide some evidence on this. The apparent significant time lapse between the project team's decision to significantly revise the performance indicators in the Results Framework in late 2007 and the formalization of this decision through a project restructuring paper in late 2010 could also have been better explained.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions