Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Nuestras Raices Program


  
1. Project Data:   
ICR Review Date Posted:
05/02/2012   
PROJ ID:
P083244
Appraisal
Actual
Project Name:
Nuestras Raices Program
Project Costs(US $M)
 16.6  15.32
Country:
Honduras
Loan/Credit (US $M)
 15.0  13.86
Sector, Major Sect.:
: General agriculture fishing and forestry sector, Primary education, Health, Other social services, General water sanitation and flood protection sector
Cofinancing (US $M)
   
Theme(s):
Participation and civic engagement (23% - P) Other human development (22% - P) Indigenous peoples (22% - P) Other social protection and risk management (22% - P) Gender (11% - S)      
L/C Numbers:
C3892
Board Approval (FY)
  2004
Partners involved
 
Closing Date
05/31/2008 07/31/2010
         
Evaluator: Panel Reviewer: ICR Review Coordinator: Division:  
April Connelly
Robert Mark Lacey IEG ICR Review 1 IEGPS1

2. Project Objectives and Components:

a. Objectives:
The objective of the Project according to the Development Credit Agreement (DCA) is "to increase the participation of the Borrower’s indigenous and afro-honduran peoples in the holistic development process of their communities." (DCA, pg. 22)

The project development objective (PDO) according to the PAD is that "indigenous and Afro-Honduran groups have increased participation in the holistic development processes of their communities and their nation. This will be achieved [through] strengthening the communities' local capacity to build sustainable organizations, development projects, and cultural identity." (PAD, pg. 4)

For purposes of assessment, IEG refers to the statement of objectives contained in the PAD because it is more monitorable.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components (or Key Conditions in the case if Adjustment Loans):
Component 1. Participatory Development Planning by Ethnic Group (estimated cost US$0.49 million, actual cost US$0.55 million) This component supported a process of participatory development planning among the indigenous and afro-Honduran groups by financing the: (a) provision of training to the indigenous groups and federations to actively participate in the local Community Development Plan (CDP); (b) provision of technical assistance to identify sub-projects; (c) processes/activities to encourage federations to work more closely with municipalities; (d) training of facilitators to assist participating communities in carrying out a needs assessment resulting in a CDP and a list of priority sub-projects; and (ii) preparation of these CDPs and sub-projects. Sub-projects were to be further prioritized in regional microplanning meetings, and subsequently, at the level of each indigenous group, using a fixed set of selection criteria (including, poverty, land tenure, and significance). Each of the nine federations was then to prepare a strategic development plan outlining the vision, mission, and goals for its membership over a three- to five-year period. Aspects of these strategic development plans were expected to feed into municipal and national development plans.
Component 2. Grants for sub-projects (estimated cost US$13.8 million, actual cost US$11.88 million) This component supported the financing of community-executed sub-projects (during appraisal it was estimated that 600-700 would be funded). Each sub-project was to average around US$20,000, with no minimum or maximum floor for financing. Resources were to be allocated proportionately among the nine indigenous groups, based upon 2001 Census data, while guaranteeing a minimum allocation for the five smallest groups. Sub-projects
were to be executed by the communities, using the community-driven development methodology. Project funds were to be transferred directly to a community-based Project Implementation Committee with an elected leadership.
Component 3. Capacity Building and Institutional Strengthening (estimated cost US$1.31 million, actual cost US$ 2.89 million ) This component supported: (i) capacity building of indigenous groups to lead their development process; (ii) institutional strengthening of the project unit (NRU) located within the Honduran Social Investment Fund (FHIS); and (iii) monitoring and evaluation. Activities to be financed under this component included training and technical assistance to communities in: (a) participatory development planning; and (b) project identification, execution, operation, and maintenance. The NRU was vested with the responsibility for managing the project in close coordination with federations, Project Implementation Committee's, community facilitators, and FHIS supervisors. This component also included operational costs (FHIS/NRU staff, recurrent costs for offices, mobilization and equipment).

In addition, the PAD estimated US$ 1.1 million for the Operational Costs for implementing the project.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost were estimated at US$16.6 million, to be financed by a US$ 15 million IDA credit and a government counterpart contribution of US$1.6 million. Actual project cost was US$15.32 million, financed by a US$13.86 million IDA credit and US$1.46 million government contribution. The ICR notes that the shortfall in disbursing the full IDA Credit was due to delays in executing the project and the lower than estimated cost of the sub-projects. The government's contribution was less than estimated due to a reduction in government budgetary revenues in 2009 and 2010 related to the international financial crisis.

The project closing date was extended three times adding a total of 26 months to implementation. The first extension was granted to compensate for delays in credit effectiveness and to allow more time for the implementation of sub-project activities. The second extension was granted to compensate for the disruption in the implementation of the Bank's portfolio caused by the sudden removal of the Zelaya government in June 2009, and to provide the newly elected government with adequate time to close project activities. The closing date was extended for a third time to allow the new administration time to complete cross-cutting projects under implementation and to carry out pesticide training in response to safeguard issues that had been identified in the final years of implementation.


3. Relevance of Objectives & Design:

Relevance of Objectives was high. The projects objectives were highly relevant to government and Bank strategies at appraisal and continued to be relevant at project completion. The project's objectives are in line with the 2001 PRSP, which covers the period of 2001 -2015, specifically the pillars of strengthening social protection for specific groups and guaranteeing the sustainability of the strategy. The objectives were consistent with three pillars of the Honduras Country Assistance Strategy (CAS), covering FY 2003-2006: (i) strengthening social protection for specific vulnerable groups; (ii) reducing rural poverty; and (iii) enhancing investment in human capital. The PDO is also consistent with the human capital development pillar of the FY07-10 CAS which calls for the continued strengthening of the capacity of afro-Honduran and indigenous peoples to manage their own development as one of its sub-strategies.

Relevance of Design was substantial. Design reflected a good understanding of the Honduran social and economic context and of the challenges faced by indigenous and Afro-Honduran communities and the project components were appropriate to achieving the PDO.

4. Achievement of Objectives (Efficacy) :

Achievement of the Overall Objective -- "indigenous and Afro-Honduran groups have increased participation in the holistic development processes of their communities and their nation [through] strengthening the communities' local capacity to build sustainable organizations, development projects, and cultural identity -- modest. The evidence presented in the ICR is related to the communities' participation in the development and implementation of sub-projects, but there is little evidence to demonstrate that the project enhanced the communities' involvement in the broader development process linked to municipal government or that the development and implementation of sub-projects has contributed to a holistic development process.

1,665 indigenous community participatory diagnoses were prepared (exceeding the target of 1,500) and incorporated into 9 Strategic Ethnic Development Plans that were completed for each of the country's nine indigenous group. However, the ICR notes (pg.21) that the indigenous plans are not "strategic" but rather a sequential list of possible investments. An evaluation carried out in the final year of the project found that the project succeeded in inducing the participation of the benefited communities in the decision-making processes related to the design and implementation of sub-projects. About 678 community facilitators, 1,655 Project Executive Committees and 12,800 community members were trained in the preparation of sub-projects. A total of 1,666 sub-projects were prepared and approved and 1,517 sub-projects were completed by project closure, greatly surpassing the target of 500.

The project met its target of having five government agencies provide partial financing for the Strategic Ethnic Development Plans but only 20 out of the expected target of 50 municipalities financed sub-projects included in the Strategic Development Plans. Moreover, the Strategic Ethnic Development Plans have not fed into municipal and national development plans as anticipated at appraisal. The KPI of "incorporating at least 400 indigenous and afro-Honduran Community Development Plans (CDPs) into Strategic Municipal Development Plans (SMDPs)" was not achieved because the FHIS could not undertake the required work with the municipalities to ensure that this happened. According to the log frame in the PAD, failure to achieve this target would “indicate that the municipal governments continue to pay little attention to ethnic communities, not giving them equal access to public resources."

As a result of participation in the project, the ICR reports (pg. 31) that an impact assessment found “most of the indigenous and Afro-Honduran populations believe that the Program helped develop their knowledge, skills, and ability to better organize themselves, administer financial resources, and make better decisions with respect to the purchase of materials used for their projects, thereby helping improve their living conditions, particularly in terms of food security and better housing. The impact assessment indicates an expected 3 percent improvement in the population’s poverty indicators, compared to control groups used for assessment purposes.” Neither baseline data nor a counterfactual were established at the project’s start, therefore the impact evaluation utilized propensity score matching to construct a counter-factual using poverty mapping statistics from the National Statistical Institute of Honduras. In addition to the community-executed sub-projects, seven horizontal sub-projects 1 were completed. However, the ICR notes (pg 21) that "in the best-case scenario, these horizontal sub-projects will only provide the civil works at the closing of the Credit, being by themselves necessary but not sufficient to induce a holistic community development process as aimed by the NR PDOs.”


Strengthening the communities' local capacity to build sustainable organizations: According to the ICR (pg. 8), an evaluation carried out at project close concluded that "as compared to the control group, the (project) did not strengthen the organizational capacity of the sampled participating communities as envisioned by the PDOs."

Strengthening the communities' local capacity to create development projects: About 678 community facilitators trained in sub-project preparation methodology and 1,655 Project Executive Committees and 12,800 community members trained in the preparation of sub-projects. 1,665 sub-projects were prepared and submitted to the FHIS for review and approval and a total of 1,517 sub-projects were completed, surpassing the target of 500.

Strengthening the communities' cultural identity : The project's M&E system did not include an appropriate metric to measure these PDOs or a baseline to measure the envisaged changes, and the ICR provides no additional evidence to assess this achievement.

1According to the project appraisal document, "horizontal projects" are cross-cutting projects that respond to cultural, patrimonial, legal, environmental, or gender problems affecting the entire ethnicity. Such projects were intended to strengthen social capital, to preserve linguistic and cultural heritage (organization of fairs and cultural events, construction of museums to house ethnic artifacts, etc.) and to address land rights issues and other issues of shared interest to the group.

5. Efficiency (not applicable to DPLs):

Efficiency is rated modest. A formal cost-benefit or cost effectiveness estimate was not carried out for the project as a whole at appraisal or project closure due to limitations in available cost and outcome data. To provide an indication of the economic benefits derived from sub-project activities, the ICR drew on the findings of an evaluation carried out in 2009 which compared a random sample of 100 participating communities with 100 non-participating communities. The evaluation measured long and short term employment generated by the project; changes in productivity; commercialization of the approved productive sub-projects outputs; and access to credit. The results indicate that the project had no impact on long term employment. A 2 to 3 percent increase in short term employment was generated in the agriculture sector, as compared to the control group, but employment was not continued once implementation of the sub-projects came to an end and the ICR does not report on the income generated. The project had no impact on improving productivity or in increasing the cultivated area for crop production sub-projects. Livestock production sub-projects produced a small output surplus (2 to 11 percent) that was sold in local markets but the income generated was marginal as communities had not received appropriate training in how to commercialize their surplus production. Access to credit was 4 percentage points higher in the project sample communities than in the control group.

Project management inefficiencies resulted from a lack of a ceiling on the number of sub-projects. The proliferation of small sub-projects generated a high transaction cost and overtaxed implementing agency's capacity to provide effective technical assistance to communities.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:



Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The project was highly relevant to government and Bank strategies at appraisal and remained relevant to strategies in place at project closure; design relevance was substantial. However, achievement of the overall objective was modest. The project increased communities' participation in the development and implementation of sub-projects at the community level but there is little evidence to demonstrate that the project enhanced the communities' involvement in the broader development process. Project efficiency was also modest.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating :

IEG concurs with the ICR’s high rating of risk to development outcome is high. The ICR noted that, by and large, the suprojects supported by the project are sustainable. The ICR data sheet reports that "between 70% and 90% of sub-projects implemented by communities" have effective sustainability plans. Some project outcomes, however, have not been sustained. For example, short-term employment generated by the project terminated once the implementation of sub-projects came to an end. As noted in the discussion of efficacy, the strategic development plans created under the project were not incorporated into the municipal planning process, so it is unclear how the federations will obtain financing to continue the implementation of their development plans following project closure. The ICR further notes that the program as a whole is not sustainable. The Honduran Social Investment Fund has no national budget allocation after October 31, 2010 and "seems unwilling to pursue any dialogue with the GOH to ensure continuation of the Nuestras Raices program". Finally, while some project areas are receiving continued support through an IBD financed operation, it is not clear what will happen to the activities in these areas once the financing of the IDB comes to an end.

a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

Quality at entry is rated moderately unsatisfactory. The project preparation team consisted of highly qualified and experienced specialists who were familiar with the preparation and implementation of the previous Social Investment Funds operations. Preparation involved engagement with the nine indigenous federations who were to benefit from the project and coordination with the IDB, which was financing complementary operations. In designing the project, the Bank team took on board lessons learned from similar Bank funded operations in Honduras and the Region, as well as the findings of an OED evaluation on prior social investment funds in Honduras. An economic analysis of sub-projects was not carried out at appraisal due to their demand-driven nature but the Bank team required the project implementation team to undertake an impact evaluation, to begin at the start of the project, to assess the cost effectiveness of the project compared to a control group.

These positive aspects in ensuring quality at entry were, however, outweighed by a number of shortcomings that had significant consequences for project implementation. Social and institutional assessments were carried out as part of project preparation to assess the political commitment of the Government to the project’s objectives and design, but the assessments overlooked important political economy issues of power struggles between ethnic federations, the communities and municipalities. The project was prepared as a repeater project. Design differed in that the earlier operation was a short-term cash for labor operation while the project under review was a community based development operation which included productive sub-projects, which the implementing agency had no prior experience in managing. In addition, there was no minimum threshold placed on financing for sub-projects. Consequently there was a proliferation of very small sub-projects that overstretched the implementing agency’s capacity to provide effective on site technical assistance and crowded out work required to bridge the gap between communities and municipalities. The pesticides safeguard was not triggered during preparation even though agricultural projects were eligible for financing, which the ICR attributes to an oversight on the part of the Bank preparation team. Finally, the ICR notes that an operational manual, including the first-year operational and procurement plan, was not available by appraisal, though it was approved six months following credit effectiveness. The project team explained to IEG that while completion of the OM by appraisal was not a requirement at the time that the project was prepared, in retrospect the team felt that this should have been an indication of the lack of readiness of the implementing agency.

Supervision is rated moderately unsatisfactory. There were three task leaders throughout the life of the project, although the ICR reports that transitions were well planned and carried out smoothly. The project was subject to thirty supervision missions carried out by a team of HQ based and local operational staff between credit effectiveness and project closure. Additional missions focused exclusively on financial management and procurement. Local Bank staff also engaged with NR staff on a weekly basis. In response to the mid-term review, which rated the project unsatisfactory in most categories, the Bank agreed upon an action plan with the implementing agency (IA) addressing implementation, staffing, fiduciary, safeguards and M&E issues. The ICR notes (pg. 5) that the IA immediately complied with the majority of actions and implementation subsequently improved. The Bank was also proactive in addressing fiduciary weaknesses in the IA (see section 11). Despite these positive aspects, supervision performance was hampered by the following shortcomings: (i) the ICR notes that most supervision missions were largely reactive in nature and the time to travel to some communities limited the amount of sub-projects that could be covered; (ii) the lack of a baseline and the delay in carrying out an impact evaluation impeded the Bank's ability to gauge systematically progress toward the PDO, though the ICR notes that visits to sub-projects provided the team with some indication of whether sub-projects were achieving results; (iii) supervision did not overcome the poor M&E design that had been inherited from appraisal (see section 10); (iv) it is not clear if the safeguard issues that arose were adequately addressed; and (v) some of the auditors’ concerns were still unaddressed at closure. See section 11.

a. Ensuring Quality-at-Entry: Moderately Unsatisfactory

b. Quality of Supervision: Moderately Unsatisfactory

c. Overall Bank Performance: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

Government Performance was unsatisfactory. The ICR reports that the government preparation team was a highly qualified project preparation team, and included the project management staff of a previous Bank funded social investment fund, key officials from relevant government ministries and representatives of participating ethnic group federations. Government support for the project was demonstrated by timely allocation of counterpart funds and political support for the social policies that underpinned the project. The preparation team took on board lessons learned through implementation of previous Bank funded social investment funds. However, there were a number of weaknesses in the prior operation that were not addressed during project preparation and which the government failed to address when they emerged during the project's implementation: the failure of cascade training to move beyond the first layer of training recipients, the absence of municipalities from project activities, signs of corruption, failure to implement a mechanism to ensure transparency and accountability, failure of previous operations to address issues such as land tenure and the lack of access to education and other basic services, the failure to sustain the impact of project activities beyond the project, lack of observable change in community behavior and failure to strengthen cultural identity. An additional shortcoming in government performance was the appointment of unqualified staff in the implementing agency. The ICR notes that the Ministry of Finance did not exert adequate pressure on the implementing agency when it failed to address some audit qualifications.

Implementing agency performance is rated unsatisfactory. The ICR acknowledges the efforts of the final two project coordination teams, which it credits with moving the project forward under a difficult political environment with limited authority and resources. However, overall, the ICR notes that the implementing agency, a unit created in the Honduran Social Investment Fund, was understaffed and its performance was adversely affected by frequent changes in both management and technical staff. The ICR also reports that there was a mismatch between the implementing agency's complex management procedures and the decentralized nature of the project. The organizational arrangement within the implementing agency was not conducive to effective fiduciary management or provision of technical support to communities. There was little pro-activity in addressing problems that arose during implementation. The ICR reports that the implementing agency did not follow up on agreements reached during supervision missions with respect to on-site monitoring, provision of technical support, monitoring and evaluation, financial management issues, and the inclusion of municipalities in the project scheme. Finally, audit reports were delayed, on average 4 to 6 months, throughout project implementation, which in turn impeded both the government and Bank team's ability to address auditors’ qualifications in a timely manner. Most audit reports were qualified, some audit observations were not followed up in a timely manner and some issues were never addressed. The implementing agency also lacked qualified expertise to supervise the implementation of safeguards. See Section 11.

a. Government Performance: Unsatisfactory

b. Implementing Agency Performance: Unsatisfactory

c. Overall Borrower Performance: Unsatisfactory

10. M&E Design, Implementation, & Utilization:

Design. A results framework was established at appraisal that included a mix of output and outcome indicators that primarily monitored process oriented aspects of the project, such as the communities' ability to prepare and implement projects. The results framework lacked an indicator to measure achievement of the specific objective of strengthening communities’ local capacity to build sustainable organizations, development projects, and cultural identity. A baseline was not established at appraisal to measure the envisaged changes.

Implementation. Project indicators were not improved during implementation. At appraisal it was envisaged that work to set up an impact evaluation would begin at the start of the project and would suggest alternative indicators to better capture progress toward PDOs, but implementation was delayed until 2009. The team considered retrofitting the PDO's, following the mid-term review in 2007, but the idea was rejected because disbursement had exceeded 82 percent, the project had consistently been rated as unsatisfactory, and the Credit was expected to close in about one more year (at the initial closing date of May 2008). Two of the project's indicators were not monitored: (i) “at least 400 indigenous and afro-Honduran Community Development Plans (CDPs) were to be incorporated into the SMDPs”; and (ii) “at least 70 percent of sub-projects were to be implemented in accordance with the established parameters of time, quality, and budget” –because an appropriate metric was not defined.

Use. The lack of baseline, the delay in carrying out the impact evaluation, and the unreliability of the FHIS’s database reportedly impeded Bank supervision missions' ability to systematically assess progress towards the achievement of PDO's.

a. M&E Quality Rating: Negligible

11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts):

Safeguards.
Environment. The project was classified as environmental category "B" and, in accordance with OP 4.01, an environmental assessment was carried out by the Government during preparation. According to the ICR (page 7), supervision missions "found that although the project provided some training to communities in preparing environmental datasheets for sub-projects: (i) the safeguard logs were generally of poor quality; (ii) the training provided was insufficient; (iii) there was a lack of supervision and environmentally-related technical assistance to the communities; and (iv) there was often a lack of qualified expertise at the FHIS/NRU to supervise the implementation of these safeguards.”
Pesticides. The ICR reports (page 7) that, due to an oversight on the part of the Bank's preparation team, the pest management safeguard (OP 4.09) was not triggered during preparation although the project was to finance agricultural sub-projects. After the Bank's supervision team discovered in March 2009 that the purchase and use of pesticides was part of some sub-projects, the Bank hired two local consultants to review all productive sub-projects and assess the magnitude of the pest management issue. They completed their work in 2009. Following this, training was provided to 335 indigenous producers in five departments on the safe and proper handling and use of pesticides, as well as alternatives to their use. According to the ICR (page 7), "the final review of the mitigation by Bank safeguards specialists was highly positive." The ICR reports that OP 4.09 was not formerly triggered during implementation as the discovery of the pesticides issue took place late in the project's life, and the triggering would have required a level 1 restructuring. The ICR concludes that "though this problem was detected relatively late, from that point forward the team was able to put a quick and effective remediation plan into place and to marshal the resources needed for its implementation" (page 7). However, the ICR does not report on any possible negative impact arising from the use of pesticides prior to 2009.
Indigenous Peoples. The ICR reports (page 7) that "the project itself was considered to be an IPDP [Indigenous Peoples Development Plan], which was consistent with the Bank’s Operational Directive 4.20. The project files attest to the indigenous people’s compliance throughout the entire implementation cycle and for the last two and a half years the Bank’s team included a specialist in indigenous issues in supervision missions."
Fiduciary.
Fiduciary and procurement management were adversely affected by the recurring turnover in managerial, technical and administrative staff that followed changes in the national administration. The ICR reports (page 6) that the Bank responded by coordinating with high level government authorities and other donors to establish a common approach to address FHIS operational weaknesses, which included: an institutional assessment of Procurement and Financial Management aspects; provision of technical assistance; preparation of an improvement plan; a workshop to discuss and analyze the institutional assessment and improvement plan and ongoing institutional improvement actions; portfolio coordination meetings among FHIS project teams; joint Bank supervision missions of all Bank projects managed by FHIS; coordination meetings with IDB to monitor the agreed common approach to FHIS issues; and repeated operational training dedicated only to FHIS staff on fiduciary requirements and practices.
However, the ICR also notes that audit reports were delayed, on average 4 to 6 months, throughout project implementation, which in turn impeded the government and Bank team's ability to address auditors’ qualifications in a timely manner. At project closure some auditors concerns were still unaddressed.


12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Risk to Development Outcome:
High
High
 
Bank Performance:
Moderately Satisfactory
Moderately Unsatisfactory
Quality at entry is rated moderately unsatisfactory, as in the ICR (based on the Harmonized Evaluation Criteria, the ICR should also have rated the overall Bank Performance MU). Supervision was also moderately unsatisfactory, due to several significant shortcomings (see Sections 8 and 11).  
Borrower Performance:
Moderately Unsatisfactory
Unsatisfactory
There were major shortcomings on the part of the government and the implementing agency that adversely affected project implementation, including failure to address financial management issues (Sections 9 and 11). 
Quality of ICR:
 
Satisfactory
 

13. Lessons:

The effectiveness of social assessments can be undermined if they fail to take into account important political economy issues at the community level.

The failure of a repeater project to take a fresh look at institutional capacity issues during project preparation can have adverse consequences for implementation.

The utility of a risk analysis may be limited if it assess individual risks in isolation of one another. Additional mitigation measures may be required in the event that all risks materialize simultaneously.

14. Assessment Recommended?

Yes
Why?
To verify the ratings and document lessons learned. To reconcile inconsistencies in subproject data reported in the ICR and assess safeguard and fiduciary compliance.

15. Comments on Quality of ICR:

Quality of the ICR is satisfactory. The ICR is thorough and provides a candid account of the projects achievements as well its shortcomings and implementation challenges. There are some inconsistencies, however, in the reported sub-project data.

a. Quality of ICR Rating: Satisfactory

(ES-Rev5-Jul/06)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions