Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Afghanistan: Emergency National Solidarity Project Ii


  
1. Project Data:   
ICR Review Date Posted:
03/13/2014   
Country:
Afghanistan
PROJ ID:
P102288
Appraisal
Actual
Project Name:
Afghanistan: Emergency National Solidarity Project Ii
Project Costs(US $M)
 526  747
L/C Number:
CH261, CH476
Loan/Credit (US $M)
 120  204
Sector Board:
Agriculture and Rural Development
Cofinancing (US $M)
 406  543
Cofinanciers:
ARTF, Japan Socal Development Fund, Denmark, Italy , France, New Zealand, Czech Republic, Switzerland, Netherlands, Cyprus
Board Approval Date
  12/07/2006
 
 
Closing Date
09/30/2009 09/30/2011
Sector(s):
General water sanitation and flood protection sector (26%), General transportation sector (21%), Other social services (20%), Irrigation and drainage (19%), Power (14%)
Theme(s):
Rural services and infrastructure (29% - P) Participation and civic engagement (29% - P) Conflict prevention and post-conflict reconstruction (28% - P) Gender (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Lauren Kelly
Kristin Hallberg Christopher David Nelson IEGPS1

2. Project Objectives and Components:

a. Objectives:
The project development objectives were to: "(i) Building on the current NSP, finance expansion to new areas to lay the foundations for strengthening of community level governance and (ii) to support community managed subprojects comprising reconstruction and development that improves access of rural communities to social and productive infrastructure and services" (Project Information Document Pg. 2).

These objectives are almost identical to the objectives found in the Financing Agreement: "The objective of the Project is to strengthen community level governance in Afghanistan and to improve the access of rural communities to social and productive infrastructure and services by supporting certain components of the Recipient’s National Solidarity Program" (Financing Agreement Pg. 4).

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
1. Block Grants for Communities (Estimated component costs were US$394m; Actual component costs were US$567m). The lion's share of the project finance was designed to provide block grants to communities to carry out sub-projects for reconstruction and development through a facilitated participatory planning process. Subprojects included community based productive and social infrastructure schemes such as irrigation, roads, water supply facilities, micro-hydropower and school buildings.


2. Community Facilitation and Sub-Project Preparation (The costs for Component 2 and 3 are reported together in the ICR. Total estimated costs for Component 2 and 3 were US$93m; Actual costs were US$108m). Component 2 supported facilitation services (29 facilitating partners) that assisted local communities develop inclusive local level institutions (community development councils), identify local development needs and priorities, develop sub-project proposals, implement, monitor and report on sub-project activities.

3. Capacity Building of Community Development Councils (The costs for Component 2 and 3 are reported together in the ICR. Total estimated costs for Component 2 and 3 were US$93m; Actual costs were US$108m). Directed through the Facilitating Partners, the project sought to build capacity of the community development councils by financing training and technical advisory services. Training included support for financial management, procurement, and technical skills. Later on, some of the more advanced facilitating partners and community development councils began to work together on gender training, institutional analysis, and social accountability audits (what has come to be known as NSP Plus).

4. Implementation and Management Support (Estimated component costs were US$36m; Actual component costs were US$72m). The project supported a project management function within the Ministry of Rural Rehabilitation and Development (MRRD) by providing financing for consultant services, equipment, operating costs and training in financial management, procurement and technical skills.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Costs and Financing. The original total estimated cost for the NSP II was US$525m including US$120m from IDA; an estimated US$244m from the Afghanistan Reconstruction Trust Fund (ARTF); an estimated US$30m from the Japan Social Development Fund, and an expected US$130m from "bilateral sources, together with other sources of funding to be identified during implementation" (Memorandum to the President, 2006 p.10-11). Actual costs were 30% greater than estimated. Total actual project costs amounted to US$747m, mainly due to the expansion of the program during the project period. Following the November 2008 supervision mission, project financing requirements were revised upwards to US$704m to (i) increase the level of finance for block grants; (ii) increase the level of finance for faciliation; (iii) and increase the level of finance for program management. Actual costs were financed with IDA (US$204m); the ARTF (US$449m); the Japanese Social Development Fund (US$10m) and by bilateral donors (US$83m) including Denmark (US$39m), Italy (US$30m), France (US$11m), and New Zealand, the Czech Republic, Switzerland, Netherlands, and Cyprus (all of whom contributed US$2m or less).

Dates. The additional financing for NSP II was accompanied by an extension of the project by 18 months, from March 31, 2010 to September 31, 2011. At the time of extension, the IDA grant was already full disbursed (the IDA grant closed on the original closing date of 9/30/2009) whereas the closing date of the ARTF grant was extended to September 30, 2011.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
NSP II's objectives are highly relevant. More than two decades of conflict had eroded the country's societal institutions and norms, destroyed its physical infrastructure, and severely reduced its provision of services. Human capital was lost through massive population flight: one third of the population had fled the country and a further 3 million were internally displaced (Stewart and Fitzgerald, 2000). The NSP program responds directly to the Government of Afghanistan's goal of restoring stability, legitimacy and trust in the State after decades of conflict by supporting a nationwide system of rural service delivery. The second phase of NSP builds on efforts to strengthen social capital and increase resilience at the village level by supporting the development of inclusive and accountable local governance structures that are equipped to identify and address pressing local development needs. This cohesion is all the more important when considering that Afghanistan is still a country in conflict and that villagers in many areas of the country must make hard choices when faced with incursions. The perception of State legitimacy - as expressed through the delivery of key basic services - contributes to stabilization efforts and thus long term development prospects for the country. The objective of supplying subprojects, or a continual stream of finance for local development needs, is a highly relevant means by which to achieve these aims (as long as they are recurrent - see Relevance of Project Design).

NSP II is the second phase of an ongoing program that has been under implementation since 2003. It is deeply rooted in the country's constitution, its Development Strategies; and the most recent Country Assistance Strategy (FY13-FY17). Its Constitution, adopted in 2004, provides a platform for state decentralization, placing responsibility for local development within elected provincial, district, and village level administrations. Article fourteen of the Constitution obligates the State, within its financial means, to design and implement effective programs to develop agriculture and animal husbandry, improve economic, social and living conditions of farmers, herders and settlers as well as nomads’ livelihoods. Afghanistan's development priorities are articulated in its 2008 Afghanistan Development Strategy which were discussed with donors during the Kabul Process in 2010 and which have resulted in a set of twenty-two National Priority Programs. NSP supports two of the five National Priority clusters, including the Agriculture and Rural Development Cluster and the Governance Cluster. The most recent Country Assistance Strategy utilizes the NSP as its "flagship" program to deliver on the second CAS Pillar of Promoting Growth of the Rural Economy and Improving Livelihoods.

b. Relevance of Design:

The relevance of the design of the second phase of NSP is rated Modest. The community driven development approach to supporting inclusive and accountable local decision-making and development planning in Afghanistan was and remains relevant. The NSP approach has filled an important gap in local service delivery. By supporting elections (implemented by various means), women's participation, and accountability of the funding streams, the approach has put in place mechanisms that support the governance objective of the program. However, there is a weak link between the means by which the second objective is conceived and achieved and the overarching goal of supporting responsive, local governance. In order to achieve the overarching governance goal of the program, funding for the community development committees would need to be recurrent and reliable. Without funding, the CDCs do not have a function. In the present configuration, under NSP III, only half of the CDCs that were receiving funding under NSP II are receiving repeater block grants. As the IE pointed out, NSP begins to have a negative impact on local governance quality when the block grants stop flowing, largely linked to the fact that expectations about what the NSP can deliver are high. The relevance of the original design declined with its rapid roll-out in NSP II - a process that strained the program's capacity to sufficiently connect with qualified technical assistance and facilitation teams knowledgeable about the warring areas. As the program expanded, the capacity of FPs to facilitate implementation and NSP to supervise and monitor was overstretched, particularly by the need to cover remote and insecure areas. The question as to the future role and institutional responsibilities of the CDCs was not directly addressed by project design, although it factored into the higher level dialogue. IEG's Country Program Evaluation of Afghanistan pointed to the need to link the CDCs to sub-national government structures.


4. Achievement of Objectives (Efficacy) :


Objective 1: NSP modestly achieved its objective of laying the foundations for strengthening of community level governance.

NSP II supported the election of 6,011 community development committees (CDCs), increasing the number of CDCs from a baseline value of 16,502 achieved under NSP I to a total of 22,513 elected councils (compared to the appraisal target of 21,600). Of the total, 22,456 CDCs had developed a community development plan by project close.

Evidence of strengthened community level governance. The impact evaluation of NSP tested a series of hypotheses which, inter alia, examine the impacts at midline and endline of NSP II on local governance. (Baseline, midline, and endline surveys were administered between August 2007 and October 2011, with the basline survey data collected 6-12 months after project commencement and endline data collected alongside project closing). It found that NSP has increased the level of appreciation of the use of democratic processes in local governance as measured by the increase in the proportion of male villagers who prefer that the village headman is subject to secret-ballot election. It found that the creation of CDCs also cause customary leaders to affiliate with representative assemblies during project implementation, and that during project implementation, there is an increase in the provision of local governance services, the activity level of customary authorities, and the role served by representative assemblies in providing local governance services. There is also strong evidence that NSP increases the provision of local governance services specific to women and that the effect persists after the block grants are disbursed. However, there is no evidence that NSP introduces new leaders into the core group of village decision-makers, and many of these impacts listed above do not persist following NSP activities. Endline data revealed that NSP was found to have a negative impact on local governance quality. After the block grants are fully disbursed, male villagers are less likely to be satisfied with the work of local leaders and are more likely to disagree with recent decisions and actions of village leaders. Complementary evidence from the VBDA indicates that the observed worsening of governance quality is most likely due to the weakening of local governance accountability structures caused by the creation of CDCs in parallel to existing customary institutions and the lack of a clear delineation of institutional responsibilities following project completion.

A caveat on the methodology. Much of the evidence in the independent evaluation of the NSP points to an uptick in performance at midline, but only weak evidence, or no impact, at endline. Many of the effects of NSP were also found to be non-durable at endline. Given the nature of the NSP program – one would expect both perceptions and performance to be at their highest levels while the block grants are flowing – and lowest if and when funding has ceased. An effective comparison, therefore, can only be made by comparing villages that experienced the same funding availability and same funding cycles. Some villages for example, only received NSP funding and only received NSP funding once. Other villages received funds from multiple donors, and in the best case scenarios, were working with facilitating partners to bridge funding gaps in the absence of the availability of NSP block grants. In this regard, it would also be important to know when the endline data was collected. For example, if endline data was collected prior to a village entering into a second or third funding cycle, then expectations and perceptions of performance would be expected to be lower. So, both more information about the status of the CDCs and their connection to funding outside of NSP and the points at which information was gathered in relation to the finding cycles would be needed to better understand the implications of the IE findings.

Objective 2: NSP II substantially achieved its objective of improving access of rural communities to social and productive infrastructure sectors.

NSP II financed 10,083 subprojects. In volume terms, four subproject categories accounted for 92 percent of total funding. These subcategories included irrigation (28%), Water Supply and Sanitation (27%),Transport (22%), and power (15 percent). Other categories of funding included rural development (5% ),education (2%), livelihoods (<1% ), and health ( <1%). Both the ICR and the Independent Evaluation provide evidence that the NSP improved access to services, The project exceeded most of its quantitative targets that it set at appraisal with regard to the number of persons reached by the program's sub-projects (See Table 2 of the ICR).

The Independent Evaluation reports on a wider dataset collected to measure "improved access", including usage of the service or utility, and the amount of time saved. In contrast with the ICR, the IE considers and reports on each subproject separately, highlighting the relative achievements across the subproject types chosen. For example, the IE found that NSP-funded drinking water projects increased access to clean drinking water, with the program resulting in a higher usage of protected sources after the block grants were disbursed. Drinking water projects also reduced the time that households spent collecting water. Power, or electricity projects, were found to substantially boost electricity use, which rose by a quarter on account of the program. NSP funded subprojects were also found to increase access to education, healthcare, and counseling services for women. NSP supported transport projects, however, only modestly increased village accessibility and there was no evidence that such projects decreased the time or increased the frequency by which male villagers visited the district center. Irrigation projects fared the worst. The IE found that these projects have no noticeable impact on the ability of land-holding villagers generally to access sufficient irrigation. In sum, it is important to consider the varying levels of effectiveness of the different subproject choices, since many - but not all- offered improved access for communities to rural services.

The quality of the subprojects is discussed in the Section entitled "Risk to Development Outcome" since the quality of the subproject designs, their construction, the appropriateness of the technology used, and the availability of funds for operations and maintenance all pose real risks with regard to the sustainability of the services and utilities delivered.

Impacts on Economic Activity. The Independent Evaluation went beyond the measurement of the project objectives and assessed the contributions of the subprojects to economic perceptions (welfare gains expected due to the project) and actual effects on economic welfare. It found that overall, the impacts of NSP on economic welfare appear to be driven more by the infusion of block grant resources than by broader impacts of completed projects on economic activity. While there were changes in economic perceptions, especially at midline and especially with women, few impacts were observed on objective measures of economic activity. At midline, there is weak evidence that NSP induces small increases in the diversity of household income sources and in caloric intake, although there is no conclusive evidence to indicate that these impacts persist beyond project completion. At endline, there is only weak evidence of impact on the amount borrowed by households. NSP has no conclusive impacts at midline or endline on income levels, income regularity, consumption levels, consumption allocations, assets, or food insecurity. There is also no evidence that NSP impacts general production and marketing outcomes. NSP does not affect agricultural yields, productivity, or harvests sales, but induces a fleeting increase at midline in agricultural sales revenue. NSP also does not affect whether households sell animals or animal products or the revenue derived from such. While NSP increases handicraft sales and sales revenue at midline, these impacts are not durable. There is, however, some evidence that NSP reduces out-migration from villages at midline or endline.


5. Efficiency:


Efficiency is rated Substantial.

Cost Effectiveness Analysis. a rural service delivery program, delivery costs - including capacity building and facilitation - were 33 percent of total project costs in NSP I. As NSP consolidated its activities and procedures in NSP II, delivery costs were reduced to 24 percent of total project costs. This measure of cost efficiency is reported positively in the ICR. However, supervision records also point to inefficiencies with regard to the cost reductions - mainly achieved by mandating shorter facilitation cycles and setting a fixed price for facilitation, regardless of the
context in which they were working (level of remoteness, areas more prone to conflict etc). Facilitating partners were often not able to deliver on the prescribed cycle and the uniform rate caused some tension between some facilitating partners and NSP management.

Another measure of cost efficiency includes comparing the unit costs for small scale infrastructure development to other like programs. Several studies have compared the unit cost in infrastructure construction for provision of basic services under NSP to other infrastructure construction modalities. The studies show that NSP attains substantial cost savings: alternative methods for constructing the same infrastructure have been found to be roughly 45 percent more expensive than NSP. Studies have specifically compared the unit costs of providing tertiary roads, for example, and have found that NSP's unit costs per km were US$7,900 compared to US$35,000 per km in the National Emergency Rural Access Project. The difference is explained by the required community participation in overseeing the construction contracts and implementation of works. However, as discussed in other parts of this review, while the costs may have been lower, an adequate comparison could only be made if the level of quality of the services and utilities provided are also taken into account.

Economic Rate of Return. Three-quarters of the project finance was spent on the community block grants: total project expenditures were US$747m of which US$567 was disbursed in support of the block grants. Three sectors - irrigation, water and sanitation, transport - represented over three-quarters (77 percent) of the total financing for the block grants. The ICR calculated a weighted average economic rate of return for the subprojects supported by the project. The project (subprojects) reportedly achieved an average ERR of 69.3% as compared to the 38.2% average ERR estimated at appraisal. The average ERR is based on a sample of 84 subprojects, including a weighted sample of the most prevalent type of project types (25 irrigation projects, 25 water and sanitation projects, 24 transport projects, and 10 power subprojects). Irrigation projects were estimated to have achieved an average ERR of 98.4%, water supply achieved an average ERR of 65.1%, power achieved a 59.8% average ERR, and transport achieved an ERR of 36.7%.

But these measurements contrast the IE's findings with regard to economic activity generated by the subprojects. The IE found that few impacts are observed on objective measures of economic activity. For example, whereas irrigation projects were estimated to have achieved an average ERR of 98.4% (as reported by the ICR), the IE found that NSP does not affect agricultural yields, productivity, or harvests sales (but induces a fleeting increase at midline in agricultural sales revenue).

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
38.2%
75%
ICR estimate:
Yes
69.3%
75%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

A highly relevant objective, program design suffered from over-extension due to demand by the Government to promote nationwide coverage of the NSP model and a failure to evolve in response to learning that was taking place throughout the implementation phase and between the different phases. NSP II achieved its goal of increasing access to services, however the Impact Evaluation points to only modest results in achieving strengthened local governance over the long term. Efficiency was substantial, with gains achieved in the area of delivery costs, although there were detectable trade-offs due to some of the program's cost-cutting measures, especially with regard to the facilitation process. Economic rates of return for the subprojects are high due to the fact that much of the productive infrastructure was rehabilitated or built after being destroyed during decades of conflict. The ongoing program could benefit from a reflection of the consideration of the IE's assessments of the overall economic impacts of the subprojects categories.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:


Institutional Risks. The role of the Community Development Councils remains unclear, and this threatens to erode many of the programmatic gains made with regard to building capacity for local governance, local decision-making and local development planning. Gains made through the CDC bylaws, especially with regard to (1) the electoral process and (2) requirements for women's participation stand to be rolled back if the CDCs are not sustained. While the CDCs face many constraints, with regard to capacity, legitimacy and their standing alongside customary authorities, they have been the main vehicle for rural service delivery for the past decade and the only link between communities and the Government. Only half of the CDCs that received funding under NSP II are slotted to receive repeater block grants under NSP III. As stated in the ICR, "without the means to undertake the functions for which they were established, it is unlikely that the remaining CDCs will be sustainable." IEG's Country Program Evaluation of Afghanistan pointed to the need to link the CDCs to sub-national government structures. As the IE pointed out, NSP begins to have a negative impact on local governance quality when the block grants stop flowing, largely linked to the fact that expectations about what the NSP can deliver are high.

Financial Risks. The importance of aligning the CDCs with subnational governance institutions is critically important given the external funding environment. The NSP Program has been funded solely with donor finance (and community contributions). With the withdrawal of most international security forces, there is likely to be a decline in Official Development Assistance. To date, the NSP program has utilized roughly 20% of the national development budget, a configuration unlikely to be sustained as donor funds are drawn down. Likewise, the current wage bill of US$24 million for 800 contract staff, is unlikely to be able to be absorbed in the national budget. Current domestic revenue levels are able to finance about 10 percent of the national budget.

O&M Risks. There is also a high risk that the rural community infrastructure may not be sustained. In many cases, there was inadequate engineering support to CDCs for sub-project design, implementation and supervision, which resulted in substandard quality works. Observations point to the use of inappropriate technologies such as diesel generators and certain types of water pumps. There were inadequate operations and maintenance plans and/or a lack of their implementation. For private works, user fees have been applied but have not been successful. for public works, the relevant sector ministries have not absorbed the operations and maintenance costs as planned.

a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:

The World Bank team should be recognized for their decision to establish and support a rigorous and functionally independent impact evaluation, and for supporting the commissioning of various qualitative studies. IEG's evaluation of the first phase of the NSP found that there was little systematic data available to assess the effectiveness of the project, and recommended that an impact evaluation be conducted in the second phase. However, a weakness at entry was the inability or decision to not build capacity for real-time learning and course correction based on interim findings from the M&E system, the impact evaluation, and several qualitative studies. The decision to roll-out the program nationwide was met with several obstacles, including a lack of adequately trained and facilitation teams knowledgeable especially about the warring areas of Afghanistan. A lack of technical expertise often led to communities opting for services which may not have been a priority, but for which there was technical advice available. Another feature, which sought to decrease project delivery costs in the second phase as compared to the first by shortening the facilitation cycle and applying a universal payment, failed to take into account the heterogeneity of the project areas (security, remoteness etc.) and caused tensions between some facilitating partners and the NSP. The decision to support the creation of additional CDCs while failing to institutionally assess and award better performing CDCs supported undermined the investment that had been made and the development of social capitol in many areas during the first phase.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

Supervision early on was highly focused on results and on integrating lessons learned from other CDD programs, and from the NSP implementation experience. The Bank Supervision team conducted a workshop in September 2007 to share World Bank experiences on fiduciary oversight and livelihood development in other successful CDD projects financed by the Bank. The workshop focused on ways to enhance and simplify the fiduciary aspects of the NSP, the future role of Community Development Councils, (CDCs) and on improving rural livelihoods in Afghanistan. The mission team also analyzed both programmatic and operational issues of the NSP that were hindering its implementation performance and development effectiveness, and agreed remedial measures and actions to address those issues. Subsequent supervision missions reported on the uptake of these recommended measures. During Midterm, the World Bank supervision team was able to keep the project on track, by suggesting that NSP not further over-extend itself, in despite of pressure to significantly expand beyond the original target communities. The NSP 2 Technical Annex planned for a rollout to 4,318 villages, for a total of 21,600 communities reached by the end of the second phase. During the MTR, NSP expressed an interest in covering 32,800 communities by the end of the second phase. The MTR team reached an agreement to limit the roll-out to 23,800 communities, pointing to the financial and operational constraints facing the problem. The Financial Agreement was adjusted to include these 700 additional communities - which were the remaining communities in districts close to complete coverage. The Bank recognized that these communities needed to be covered to avoid social tension between villages, but that a wider roll-out would jeopardize the effectiveness of the program.

The security situation deteriorated precipitously during the project period, affecting the ability of the project to achieve intended impacts in some of the conflict affected areas. Some facilitating partners were forced to suspend implementation due to security delays. The Bank was responsive. It helped to formulate an engagement strategy for working in the high risk areas that was piloted in certain districts of Kandahar province. The Bank provided guidance on adopting formal risk management tools so that the NSP Management team could identify, evaluate and determine strategies for managing risks and uncertainties. This recommendation - to help project teams identify the level of risk that the Government and the World bank are willing to take - and to develop a set contingencies and performance benchmarks in line with those risks - highly informed one of IEG's key recommendations in the FCS Evaluation (2013) with regard to identifying and preparing for risks in the FCS context.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The government has been a strong champion of the NSP program. It has facilitated strengthened community level governance and has directed critical donor funding - about 20% of the national development budget - towards basic service delivery in the country's rural areas, including areas severely affected by the war.
Throughout the project, the government of Afghanistan was responsive to project needs, resolving issues related to the disbursements of community block grants through the Afghanistan Bank. Project funding however was perennially a problem for this project: the Afghanistan Reconstruction Trust Fund was under-used by donors, a portion of donor aid for rural rehabilitation and development was being directed outside of the CDC system, and in despite of guidance provided by the Bank early-on, the program lacked a strategic and timely fund-raising strategy. The role of the CDCs is also lacking strategic long-term commitment by the Government of Afghanistan, a situation that runs the risk of reversing a decade of heavy capital and human investment (both in terms of financing and lives lost). Although a proposal has been put forth by four key Ministries (including MRRD and the Independent Directorate of Local Governance) to formally recognize the CDCs as Village Councils - and for all Ministries to channel their community level development activities through these Village Councils - the proposal has not been formally accepted by the President's Office, rendering the future status of CDCs uncertain.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

This program was implemented by the NSP unit within the Ministry of Rural Rehabilitation and Development (MRRD) and executed by 29 Facilitating Partners (NGOs) at the site level. MRRD has demonstrated a high level of leadership and commitment to this project throughout the entire project cycle. The Ministry of Rural Rehabilitation and Development took on increased responsibilities under NSP II. Whereas in the first phase, international staff oversaw the program regionally, during the second phase, national MRRD staff assumed roles as regional directors of NSP II. During the transition, there were some concerns over quality and speed of procurement, notably with the recruitment of facilitating partners which were delayed by several months. MRRD demonstrated strengthened capacity in areas such as budget forecasting, resource mobilization, strong coordination with the facilitating partners, and a commitment towards rigorous and independent assessment of the program's impact (even in the face of political obstacles). MRRD staff were not always able to address financial management issues however in a timely way - an issue that project supervision was highly attuned to. The Ministry lacked a system to adequately manage Safeguards, especially land acquisition/access to land, and provided inadequate training to local staff on procurement rules.

There were also 29 Facilitating Partners that carried-out the day to day operations with the CDCs on the ground. The Facilitating Partners were extremely dedicated to this program and were instrumental in mobilizing communities, particularly in the crisis areas of the country. As the security situation deteriorated, operations in 935 communities were suspended, as several dozen people working on NSP (NSP and FP staff, CDC members), were killed, injured or kidnapped. Many FPs also pre-financed NSP activities during funding gaps or delays.

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The project supported an M&E department within the NSP office of the Ministry of Rural Rehabilitation and Development. This department was tasked with overseeing all data collection conducted by the facilitating partners, analyzing it and reporting findings to the Ministry of Rural Rehabilitation and Development. Per the Bank's recommendation, the M&E system was linked to a web-based MIS which was being developed simultaneously. The M&E system in NSP was a strong improvement over the M&E system developed for NSP I. However, key indicators developed by the project were insufficient to measure the first project objective of supporting strengthened local governance. The indicators were limited to outputs, including (1) a target number of CDCs established (democratically) across the country; and (2) a percentage of CDCs that are expected to be functioning in order to address critical development needs as identified by villagers. As demonstrated by a parallel impact evaluation managed by the task team, the indicators should have been developed in manner that measures perceptions and that test whether the project interventions in support of the CDCs have indeed been foundational. Such indicators could have attempted to measure whether social capitol was strengthened (trust, collective action,participation) and whether local governance was enhanced through local leadership transformation, increased satisfaction with local development planning and responsiveness to local needs. The M&E system could have attempted to measure the spill-over effects in neighboring non-NSP villages. A key question for M&E would have been whether stakeholders perceive that the CDCs will maintain a governance and service delivery role in the absence of project funding. The indicators developed to measure improved access of rural communities to social and productive infrastructure and services were also weak. The indicator most directly linked to this objective asks the project to report on whether operations and maintenance (O&M) is in place for completed supprojects - and that the infrastructure services are used appropriately by the targeted communities for the purposes intended. The indicator does not directly measure access - for whom and by whom - and whether for any particular taget group - this access was improved. As demonstrated by the IE, useful indicators could include an assessment of usage, perceived utility, and/or the amount of time saved in relation to the incremental access afforded by the new or improved service delivered.

The project also supported an independent multi-year randomized control trial designed to measure the effects of implementation of the second phase of NSP on a broad range of economic, political, and social indicators. While there have been a number of qualitative studies of NSP, the NSP-IE is the first large-sample quantitative assessment capable of providing rigorous estimates of program impact. Baseline, midline, and endline surveys were administered between August 2007 and October 2011. Collectively, the surveys comprised over 25,000 household interviews with male and female villagers, as well as more than 2,600 focus groups with male village leaders and women. The IE also drew on a village benefit distribution analysis (VBDA), which assessed program impacts on the equity of food aid distributions by village leaders. In addition, NSP-II supported a broad suite of research undertakings and studies (both qualitative and quantitative). According to interviews with TTLs of the program, this was a dramatic improvement over what had been the case with NSP-I. However, there are ascpects of the IE design that could have been improved, including controlling for "like-CDCs" with regard to their funding sources and patterns.

b. M&E Implementation:

NSP II's M&E system mainly focused on inputs, processes, and output reporting, and less on on the assessment of program outcomes. It was, for the most part, unidirectional: information was collected by the M&E unit, analyzed and delivered to NSP Management in the Ministry, but processes were lacking for (1) sharing information between the 20 Facilitating Partners and for (2) redirecting analytical findings downwards (from the Ministry, through the FPs, to the communities). Similarly, the M&E system lacked a mechanism to incorporate the interim findings from the IE, and/or the qualitative assessments that were being undertaken alongside the IE. Processes were also lacking for sharing learning between different donors invested in rural development, peace and security in Afghanistan. Key facets of the program that were underanlayzed included the feasibility of the different subproject choices and the varying levels effectiveness of the CDCs. One of the FPs - the Aga Khan Foundation - has since adopted NSP Plus which put in place new metrics including an Institutional Analysis and a Gender Analysis.

a. M&E Utilization:

In addition to the independent evaluation, NSP-II made great strides in developing and supporting a broad suite of research undertakings and studies (both qualitative and quantitative) as compared to NSP I. However, there was a disconnect in synthesizing the findings and operational implications of such research and feeding it back into the program in the form of changes to the design or implementation modalities of NSP. For example, one of the studies built into the IE on methods of sub-project selection provided fairly precise policy recommendations to NSP that have not since been taken up in NSP III.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

Bank supervision early in the project implementation period pointed to the lack of proper implementation of the World Bank's safeguards. According to Bank supervision records, there was lack of compliance guiding land acquisition, in particular monitoring voluntary land donations. According to the ICR, the project Management Information System did not contain sufficient records on how land was obtained or how environmental impacts were managed. This was a complex project whereby all entities (the NSP Management Unit, the 29 FPs, and all of the CDCs) had to put in place measures to ensure WB safeguard compliance. Yet the project did not have a full time safeguard specialist until 2009. Although FPs were required to designate safeguard focal point officers, only a few had them in place. Although the ICR states that the subprojects financed created no significant negative environmental or social impact, it does not provide the evidence to substantiate this claim, especially in light of the fact that the project Management Information System did not collect or report on this information.

b. Fiduciary Compliance:

Financial Management and Fiduciary Compliance. The project financial management systems, procedures and controls were generally satisfactory. NSP project grant accounts and the processing of payments were managed at the national level by a Specialized Disbursement Unit within the Ministry of Finance. At the project level, an international firm was employed by MRRD for financial management and block grant management. A Float Account was opened at the Afghanistan's Central Bank for disbursement of the block grants to CDCs. One weakness of the financial management of the program pertains to the reconciliation of IDA/ARTF grant advances to the Float account since the MRRD was using the Float accounts for other funds as well (bilteral donors). This practice has since been discontinued and the float accounts under NSP III are only being used for NSP III. Annual audited financial statements and audit reports were submitted regularly. The auditor opinion was qualified for the first two years and unqualified for the following two years. The MRRD has responded to the audit findings satisfactorily (ICR p. 12). There was one incident of misappropriation of US$3.2 million by a Hawala dealer engaged by MRRD to deliver cash to communities in the insecure areas of Paktika province. The government has since made the refund and the use of Hawala dealers has been stopped.

Procurement. Procurement was a challenge throughout NSP II. The program utilized an international firm for the first half of the project, but failed to effectively train local staff. National staff, once trained, tended to pursue more lucrative offers with the Government and within the donor community. Many of the procurement challenges faced by NSP II relate to the challenges of working generally in fragile and conflict affected states. NSP II had difficulty getting the right expertise and quality goods in local markets; costs exceeded estimated costs in the procurement plans; and service providers often reneged on contractual obligations. A lack of training and retention of local staff with knowledge of the Bank's Guidelines resulted in the delay of the selection of facilitating partners for up to 16 weeks, and a delay in disbursements. Procurement was a particular challenge in the high risk and remote areas of Afghanistan.

c. Unintended Impacts (positive or negative):
Gender. The Independent Evaluation found that NSP impacts the economic perceptions and optimism of villagers, particularly women. Female villagers exhibited improved perceptions of the current economic situation and were more optimistic than men, both at midline and endline. The economic perceptions and optimism of male villagers improved at midline, but there was only weak evidence of an impact at endline on optimism and no evidence of a longer-term impact on perceptions.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
NSP II achieved its goal of increasing access to services, however evidence points to modest gains in achieving strengthened local governance, due to uncertainties about the financing and institutional role of the CDCs over the long term.  
Risk to Development Outcome:
Significant
High
Withdrawl of Stabilization forces is affecting the levels of donor finance that is critically needed to reenforce the CDCs whose institutional role remains uncertain. Evidence points to decreasing trust in the function as and when funds become unavailable. Gains made in representation and participation will likely be eroded in the absence of the NSP model. Low quality of technical plans for works also draws into question the sustainability of the services delivered.  
Bank Performance:
Satisfactory
Moderately Satisfactory
Some implementation challenges can be traced to weakness in design; the project lacked a mechanism for course correction in despite of the fact that is has invested heavily in evaluation.  
Borrower Performance:
Satisfactory
Moderately Satisfactory
Due to the lack of strategic planning on the role of CDCs, their institutional alignment and the future sustainability of services - the effects of discontinuity of this program are magnified in an FCS context, since the delivery of services is intricately tied to the perception of state legitimacy and brokering peace.  
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
ICRR Lessons are derived from the ICR, interviews with two of the project TTLs and Lead Impact Evaluator, findings from the independent Impact Evaluation of NSP II conducted between 2006-2011, and the findings and recommendations from IEG's Country Program Evaluation of Afghanistan (2002-2011). Since the NSP is an ongoing program, and the findings and lessons of the impact evaluation were not presented in the ICR, this ICRR presents lessons that are specific to the program, that can also inform Bank supported CDD programs elsewhere.

There is a need to advise and support the Afghanistan government in transforming the NSP into a more sustainable financial and institutional model to consolidate its gains. This would involve helping to formulate a long-term strategy around the future role of the CDCs. It would involve supporting a dialogue around the relative effectiveness of linking the CDCs to higher tiers of subnational governance, including the tradeoffs associated with transition in the short-term versus the longer term need for regular intergovernmental fiscal transfers for public investments and service delivery.

The design of the NSP program has not evolved since its initial start-up phases. There has been a dramatic improvement in data collection, monitoring and evaluation over the lifetime of this program. However, to date, there has been a disconnect between the collection of project level data and the synthesis and incorporation of this data into program design. The program would benefit from a deep consideration of the operational implications of findings that have emerged from the Impact Evaluation and other qualitative studies on subproject selection, gender, elections etc.

Building not just capacity but institutional legitimacy is extremely important. In light of the
low level of institutional depth both in line Ministries and at the local level, considerable technical
assistance has been provided to build capacity. In providing such capacity, what has succeeded is when
Bank support has helped the Government build institutions, systems and social capital that are seen as
legitimate because they are inclusive and provide services.

While the program has made exceptional progress on promoting women's participation in decision making at the local level, the menu of options with regard to women's participation may have been too rigid, suggesting that more women may have participated with a more flexible and intuitive approach. This would require increasing the number of female facilitating partner social organizers, more extensive gender training, and finding ways to build more local ownership into the initiative- taking into account the very different attitudes and perceptions about women across the country's heterogenous provinces. Within the CDCs that have incorporated women's participation, more extensive research is needed on how this participation i has impacted villagers' perceptions about the role of women in society, how this participation has influenced decision-making and sub-project selection, and how women's participation in the CDCs has influenced women's participation outside of the CDCs.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is rated Satisfactory, with shortcomings. The ICR is comprehensive, covering all requisite categories. It was forthcoming about issues related to procurement, financial management, and to some extent, on Safeguards. However, having invested a significant level of funding and resources in the conduct of the IE, it is not clear why the findings of the IE were not presented nor discussed by the ICR. The ICR (Footnote 7 Page 29) refers to the Baseline Surveys as having been conducted in summer 2007; midline in 2009; and endline in 2011. The ICR is dated May 2012. The ICR assessment of governance, for example, diverges from the IE findings, and lacks substantiation. Likewise, findings from key qualitative studies that had been conducted on aspects of project implementation were not reported. The ICR provided information on project outputs (subprojects) - but lacked a discussion of the varying degrees of benefits afforded by the different subprojects (public versus private goods) as they relate to achieving the ultimate goal of sub-national governance. Ratings were not always realistic. The ICR mentions several design and implementation challenges, but these are not reflected in the ratings of Relevance (of design) or Bank Performance.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions