Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Emergency Social Safety Net Enhancement (ec Food Facility Grant)


  
1. Project Data:   
ICR Review Date Posted:
06/27/2013   
Country:
Yemen
PROJ ID:
P117038
Appraisal
Actual
Project Name:
Emergency Social Safety Net Enhancement (ec Food Facility Grant)
Project Costs(US $M)
 23.3  23.3
L/C Number:
Loan/Credit (US $M)
 23.3  23.3
Sector Board:
Social Protection
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  12/16/2009
 
 
Closing Date
06/30/2011 06/30/2012
Sector(s):
Other social services (100%)
Theme(s):
Global food crisis response (100%)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Xue Li
Rene I. Vandendries Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The project development objective is identical in the European Union Food Crisis Rapid Response Facility Grant Agreement (p.7) and the Emergency Project Paper (p.8), namely “to contribute to the reduction of the negative impact of food price volatility on the poor and vulnerable in selected areas, and support the protection and building of community assets in poor communities.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

The ICR (p.6) identifies two Components, consistent with the Project Paper (p.8-9) and the Grant Agreement (p.7), and attaches costs to each of these two components.

Component 1: Workfare Assistance and Impact Evaluation (Euro 10.197 million)

Sub-Component 1.1. Expand workfare assistance for the needy (Euro 9.8 million). This sub-component was to finance the provision of sub-grants under the workfare assistance program to about 12,000 to 16,000 households within communities most seriously affected by the food crisis, for the carrying out of basic infrastructure activities (subprojects).

Sub-Component 1.2. Impact Evaluation and component management – workfare program (Euro 0.397 million). This sub-component would support the implementation of the program’s phase II baseline study, and an impact evaluation of the workfare assistance program.

Component 2: Temporary Cash Assistance for Poor Households (Euro 7.303 million)

Sub-Component 2.1. Cash transfers to poor and vulnerable households (Euro 7.140 million). This sub-component was designed to provide temporary cash assistance grants to about 41,000 eligible poor households in flood, conflict or child trafficking areas. Those households would receive US$ 20 equivalent/month to help them cope with the price pressures over 12 months.

Sub-Component 2.2. Evaluation and component management – cash assistance program (Euro 0.163 million). This sub-component was designed to support an evaluation of the temporary cash assistance program to address the targeting, operational processes and results of the program at the household level.

Component 1 - Labor Intensive Work Program (LIWP) was implemented by the Social Fund for Development (SFD). Component 2 was implemented by the Social Welfare Fund (SWF).

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

The project was funded by a grant of US$23.3 million (at appraisal) from the Global Food Crisis Response Program (GFRP) European Union Food Crisis Rapid Response Facility Trust Fund. Funds were fully disbursed for Component 1 and Sub-Component 2.1, while only 57% of funds were disbursed for Component 2.2 – Evaluation and component management – cash assistance program. The supervision budget was supplemented by the government and part of the evaluation (the analysis) was funded through another resource as the evaluation could not be completed before the project closing due to security constraints at that time.

The Project was approved on December 16, 2009. The Grant Agreement was signed on March 16, 2010, and declared effective on April 27, 2010.

The Grant’s Closing Date was extended twice, through project restructuring. The first extension was to extend the original Closing Date of June 30, 2011 to December 31, 2011. On December 7, 2011, the Government of Yemen (GOY) requested a second extension until June 30, 2012. At that time, the Project was under suspension due to the prevailing countrywide suspension of disbursements during the period of intense internal conflict. However, according to BP 13.30, paragraph 9, the Bank delayed the closure of the Grant Account and Cancellation of the Grant Funds until the suspension was lifted on January 20, 2012. On March 1, 2012, the Grant’s Closing Date was retroactively extended until June 30, 2012.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

The project’s objectives were highly relevant at appraisal and on completion. At appraisal, Yemen was among the poorest countries in the world. The country’s fiscal revenues, heavily dependent on the oil sector, declined dramatically in 2009, leading to significant cuts in the public budget to most sectors, including the social sectors. In addition, the rise in international food grain prices in the context of meager per capita income growth was seen as having the potential to reverse achievements in the reduction of poverty over the previous seven years.

The objectives are relevant to the Government’s Development Plan for Poverty Reduction (2006-2010) that aimed to increase the coverage and effectiveness of the targeted cash-based safety net to address the impact of food prices and other shocks in the country. More importantly, the project objective is fully aligned with the Transitional Program for Stabilization and Development (TPSD) for 2012-2014 launched by the Government of National Unity of the Republic of Yemen in June 2012. The TPSD proposed parallel implementation of an Emergency Response and the initiation of a medium-term Economic Recovery Program.

Project objectives are also relevant to the current (2010-2013) Country Assistance Strategy (CAS)’s third pillar that aims to foster human and social development, and that pillar’s third thematic area to “expand the umbrella of social protection and services to poor communities.” This project is also aligned with the Bank’s Strategy Update for MNA, that underlines the need to improve governance through transparency and accountability measures, create jobs - particularly for youth and the poor, and increase social and economic inclusion of disadvantaged groups.

Relevance of Objectives: High

b. Relevance of Design:

The relevance of design is substantial, and the project’s results chain in the Project Paper is plausible. Since this project provided an expansion of the emergency Additional Financing (AF) Grant under the GFRP, it adopted the same basic design for both the workfare and the cash transfer programs. The workfare assistance component would reach an additional of 12,000-16,000 households within communities most seriously affected by the food crisis. The temporary cash assistance component would reach an additional 41,000 poor households in flood, conflict, and child-trafficking areas. For the workfare program, cash will be provided by offering labor intensive community works (inputs), thus will create temporary work opportunities (outputs) in poor communities. Under both programs, households would use the cash assistance to buy food and pay off debts, which would help reduce the food gap, decrease debt and protect productive assets (intermediate outcome). As a result, the negative impact of food price volatility on households’ consumption and expenditure would be mitigated (outcome). At the same time, the labor intensive community works will be identified based on the demand and priority needs of each community. When these subprojects finish (inputs), communities will benefit from better basic infrastructure such as protected soil, rehabilitated agricultural terraces, and improved local feeder roads (outputs).

In terms of project design with respect to the payment frequency of cash assistance under Component 2, unlike other countries’ experience of monthly or bi-monthly payment, the cash assistance was paid as a three-month lump sum, which is consistent with the government’s cash transfer program. Although the quarterly lump sum payment raises some questions on whether this was the most effective way to proceed, the administration cost of monthly transfers of $10-$20 per household would be too high especially on the part of the beneficiaries. For example, those living in remote rural areas have to travel long distances or pay people to collect the cash. In some parts of the country, post office staff have to use motor bikes to carry the cash to remote locations where the beneficiaries cannot travel to the postal branch offices for such small amount.

Relevance of Design: Substantial


4. Achievement of Objectives (Efficacy) :

There were two main objectives:

(1) To contribute to the reduction of the negative impact of food price volatility on the poor and vulnerable in selected areas: Substantial
Outputs:
Workfare assistance: areas mostly affected by drought, floods, and food price volatility.
A total of 102 Labor Intensive Work Program (LIWP) subprojects were completed from the initial 136 communities that were targeted, creating employment for 17,818 households (against targets of 100 subprojects and 12,000 households, respectively).
A total of 964,211 employment days (against a target of 800,000 days) were created as a result of the implementation of the 102 LIWP subprojects.
The average duration of employment offered to each household was approximately 50 days (against a target of 66 days), implying an average transfer of around US$544 per household, over an average period of 4-6 months.
Female participation rate in the program was 14.4 percent.
Temporary cash assistance: poor households in flood and conflict affected areas, and areas with high prevalence of child trafficking.
A total of 40,441 newly enrolled households benefited from the SWF’s direct cash transfer program (against a target of 41,000 households).
29 percent were female-headed households (against a target of 10-20 percent).
Proxy Means Test (PMT) was applied to determine participation for all cash transfer beneficiaries.
Cash received equaled US$20 per month over 12 months.
99.4% of households collected their benefits on a timely basis (against a target of 100%).
Outcomes:
Workfare assistance:
(i) Employment generation: findings from the Impact Evaluation show that the LIWP
Increased the total number of days worked per household by approximately 60 days per year,
Increased average wages and shifted the structure of the workforce away from work in the lowest paid sectors, and
Caused a significant increase in the probability of female employment.
(ii) Food security and livelihood outcomes: statistically significant program effects on food consumption, debt repayment, and durable goods ownership.
On average, households in treatment communities had increased per capita calorie consumption of staple foods relative to a control group by approximately 300 calories per day, and were less likely to self-report that adults were forced to skip meals due to food shortage.
Among indebted households, households in treatment communities were able to pay off more debt, by 30,000 Riyals (US$141) on average.
Households in treatment communities also had less decrease in the value of durable goods owned.

Besides the above encouraging results, the Social Fund for Development (SFD) Impact Evaluation study also revealed an uneven distribution of benefits among households, ranging from as low as US$235 for 29 percent of households to as high as US$1,415 for 5 percent of households. This uneven distribution of benefits was a result of several factors: (1) although the LIWP was designed to be self-targeting by setting a wage lower than the prevailing unskilled wage in the areas, due to the economic crisis, LIWP employment became more attractive than originally designed; (2) the original intention to limit households to a maximum number of workdays was not enforced; (3) since program wages were set by piece rate, workers involved in more skill intensive tasks or who worked longer hours got higher wages per day. Correspondingly, the number of men in the family was the strongest predictor of program participation and total benefits received.

This uneven distribution of benefits together with the low female participation rate and the types of community works raise questions on the targeting and coverage of the workfare program. On the one hand, in terms of targeting, although selected communities are indeed poor, the self-targeting wage was also attractive to non-poor households and was not adjusted by the team when this problem emerged as more households participated than planned. The respondents of a related qualitative study confirmed unanimously that participation was open to all households in targeted areas (ICR p.19). In addition, the types of subprojects implemented are all labor-intensive works which are more suitable for men than women; and there was no quota for female participation and no significant adjustments (although briefly mentioned in the ICR) to encourage women to participate, which resulted in a relatively low female participation rate. Given that female-headed households are more likely to live under the poverty line, the lack of female quota and sufficient adjustments weakened the program’s target to help more poor and vulnerable households to cope with the food price volatility. On the other hand, with regard to coverage, as more non-poor households were potentially attracted by the labor intensive works, it is possible that some indeed poor households may be crowded out, which would lower the program’s coverage of poor households in these areas.

Temporary cash assistance:

In early 2012, all beneficiary households were included in the regular SWF coverage based on the GOY 2012 budget to ensure continued transfers after Project completion.
All of the beneficiary households were located in flood and child trafficking areas.

The Project also financed an external consultancy firm to assess how well the cash transfer component targets poor households and the process of implementing the project, as well as to evaluate the impact of the cash transfer on the SWF beneficiary households. The baseline survey was carried out during September 2010 – June 2011. Since the project became effective in April 2010 and the first three months payment was already made in November 2010, it is questionable whether this baseline survey really captured the baseline situation in the treated areas.The follow up survey and the final report was not ready when the ICR was prepared. However, the TTL provided updated information as follows: The final survey confirmed earlier findings that 95% of the beneficiary households used funds for food and pay off debts. However, 83% of the beneficiaries reported that cash transfer didn't sufficiently cover their needs and 82% had to buy food on credit. In addition, 83% had insufficient food frequency and 54% reported poor dietary diversity. Main reasons would be (i) the limited benefit amount, and (ii) the extended disbursement suspension period suffered by the project.

(2) To support the protection and building of community assets in poor communities: Substantial

Outputs:
Number of subprojects fully financed was 102 (against a target of 100).
Subprojects were carried out in the fields of soil protection, agricultural terraces rehabilitation, maintenance and improvement of local feeder roads, paving of streets and other types of labor-intensive works based on the demand and priority needs of each community.
Outcomes:
In terms of the perception of the program effect on community assets, 95% of households reported that the subproject was needed by the community and 79% of households reported that they benefitted directly from the infrastructure built by the subproject.
No significant changes in measurable benefits of infrastructure constructed, such as time spent fetching water or traveling to the nearest market, were reported.

Although the assets were created and communities are using and maintaining them, it is too soon to evaluate the longer-term food security effects of local public goods constructed under the program. According to the qualitative study done by SFD, beneficiaries reported that they believed the infrastructure projects responded to their priorities, of which water continues to be the highest. Beneficiaries of nine of the 14 projects visited highlighted the importance of maintaining and protecting their new infrastructure, and five of them confirmed that maintenance committees were being formed.


5. Efficiency:


The ICR provides a brief discussion of Efficiency (p. 20-21) and a detailed “Economic and Financial Analysis” Annex (p. 33-37). Overall, both SFD and SWF were able to implement their respective components of the Project efficiently.

First, in terms of operational efficiency, SFD was able to transfer 70% of allocated funds to beneficiaries as wages, surpassing the target of 60%. The rest are cost of inputs and the overhead costs of delivering grants to households. Similarly, SWF had low operational costs in delivering cash transfers. Only $61,477 out of an allocation of slightly over $7.0 million was allocated to the institution’s overhead.

Second, in terms of the effectiveness of the targeting mechanism, the beneficiaries of the cash transfer program were selected based on the proxy means test (PMT). The beneficiary communities of the workfare program were identified based on a set of eligibility criteria. Within the selected communities, participating households were identified through a self-targeted method whereby wages were set at below the market rate. However, the self-targeted wages were more attractive than originally designed due to the economic crisis, resulting in an unexpected large number of participants, and thus weakening the effectiveness of the targeting mechanism of the workfare program.

Third, with regard to benefit incidence and poverty impact of the cash transfer and the LIWP, using the 2005-06 household survey data, the analysis shows that the Project as a whole reached about 4.6% of the Yemeni population, and 12.6% of poorest 10% of the population. For the poorest 10% of the population, the Project benefit size represented about 13% of household per capita consumption. The cost-benefit ratio that measures poverty gap reduction obtained due to the transfer is estimated at 0.72 using the national poverty line, meaning that for each €1 spent in the program, €0.72 had gone to reduction of the national poverty gap.

Taking all available evidence into account, efficiency is rated as Substantial.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
%
%
ICR estimate:
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Relevance of the development objective was high and the relevance of design was substantial. Efficacy is rated substantial for both objectives. The project has achieved all the key targets. There were 17,818 households participating in the workfare program, and 40,441 households received direct cash transfers. Around 95% of beneficiary households reported that cash assistance was used to buy food and pay off debt. As a result, households in treatment communities had increased per capita calorie consumption of staple foods relative to a control group by approximately 300 calories per day. Efficiency is also rated substantial.
Outcome is rated Satisfactory.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The risk to development outcome is rated moderate, based on the following considerations:

First, the most notable challenge for sustainability is financial, in terms of counting on continuous budget flows to sustain program expansion. The SWF is fully government funded, but the GOY faced insufficient operational budget (the share of the social sector budget decreased from 2 percent to 0.5 percent in an environment that total government budget also declined). The project was providing the same benefit amount ($10-$20 depending on the size of household) at the same intervals (quarterly lump sum) as the SWF regular program. These 40,441 households have been added to the SWF beneficiary list and are continuing to receive the same benefits as under the project. Nevertheless, given GOY budgetary constraints, sustainability may be an issue. In addition, the SWF is facing an increased need to develop and institutionalize a beneficiary recertification mechanism that will allow for filtering beneficiaries and further improving targeting, including through beneficiary graduation strategy to filter out ineligible beneficiaries to free resources. Similarly, the SFD counts on funding from about 15 donors and the government, and is also facing significant shortfalls to effectively deliver its vital services.

Second, for those subproject beneficiaries who participated in the LIWP, there is no clear evidence whether the labor intensive work has helped them gain more skills to become more competitive in the labor market, thus there is no guarantee that these people will find new jobs after this project is closed. If not, they will still face hardship due to the high and volatile food prices and the poor economy.

Finally, the capacity of both institutions will also affect the project’s longer-term outcomes. The SFD has better capacity, but the program expansion target is challenging. SFD's current plan to create 24 million person/days (from 2011-2016) may be too optimistic as they have created approximate 3.3 million person/days to date. The SWF is in continuing and greater need to strengthen its capacity, particularly in the areas of human resources management and monitoring and evaluation. Thus, it is critical to have commensurate improvement in its capacity to manage the growing needs.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:

The project is built upon the Bank’s long-term engagement in the Social Protection sector in Yemen, including both through investment lending and technical assistance. In particular, in light of the emergency need of those who are poor and vulnerable, the Bank scaled up the emergency GFRP Grant and adopted the same basic design for both the workfare and the cash transfer programs, providing timely assistance with less than six months preparation time. Since the emergency GFRP Grant is attached to a well-going Bank project with the Social Fund for Development, it provided a sound foundation for project design and expansion.

Quality-at-Entry Rating: Satisfactory

b. Quality of supervision:

The Bank’s team supervised the Grant in a very difficult environment. Supervision missions were mostly conducted in a neighboring country, but were carried out routinely in a timely manner. As the SFD had been engaged with the Bank for many years in support of their key programs, the need to conduct supervision missions outside of the country didn’t affect the Bank’s ability to engage constructively and proactively with the clients to provide operational support.

However, as noted in the ICR, the self-targeting wage for the workfare program became more attractive than originally designed due to the economic crisis, resulting in higher than expected numbers of participants and lower benefits received per household. Although this issue emerged during project implementation, the team didn’t make corresponding adjustments to address this problem.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

Despite the difficult internal environment, the Government continuously demonstrated commitment to the Project and its development objectives. Although the Government faces enormous fiscal constraints, it provided required counterpart funding, although in mid- to end-2010, it delayed its contribution to the SWF to cover for the operating cost of Component 2, causing difficulties for the Project to meet the SWF’s operating costs which were ineligible for Grant funding. Moreover, both the Government in office at the time of approval, and the Government of National Unity that took office in early 2012, considered both SFD and SWF as important institutions in their efforts to alleviate poverty.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:

On the one hand, SFD, as a more established institution, with greater experience in implementing Bank-financed projects, demonstrated strong implementation capacity. On the other hand, SWF had less experience in working with the Bank. Although weaker institutionally, especially regarding fiduciary and monitoring and evaluation capacity, SWF nevertheless was able to make significant progress towards its respective development objectives, and has worked closely with the Bank and donors on efforts to continuously improve its performance.

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

Monitoring and evaluation was designed to be carried out by both SFD and SWF, with monitoring reports submitted semi-annually as per the monitoring indicators in the Project’s Results Framework. SFD monitoring would also include participant assessments and field assessments to develop a final list of potential participating villages with high poverty rates. The SWF would monitor the number of new beneficiary enrollments from selected areas as well as determine whether the PMT was used in selecting beneficiaries. The Project’s design also contemplated an impact evaluation for both Components. The impact evaluation for Component 1 was designed to measure food consumption behaviors and changes due to project cash assistance, and community perceptions of benefits arising from protecting and building community assets. The impact evaluation for Component 2 would assess the targeting efficiency of the PMT, the efficiency of administrative and operational processes, and the impact of the program on households’ welfare.

b. M&E Implementation:

Throughout the project, SFD presented regular bi-monthly progress reports even during the crisis in 2011 to keep all stakeholders informed and accountable for resources they were using. SFD monitored its subproject implementation through its Management Information System (MIS) together with field visits by branch office staff. SWF also submitted regular semi-annual progress reports detailing progress as per the indicators in the Project’s Results Framework. And both institutions have demonstrated strong priority in monitoring the accuracy of their targeting efforts.

a. M&E Utilization:

As the ICR noted, SFD has been extremely proactive in using monitoring and evaluation results and information collected through its comprehensive MIS to improve operations and strengthen program design. As a result, SFD has adjusted and expanded the LIWP in response to the crisis, based on experience under the Project. Since the impact evaluation study for cash assistance program was not ready at the time the ICR was prepared, there is no evidence that the information gathered has been utilized for any policy or operational purposes beyond project implementation itself.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

According to the ICR (p.13), the safeguard policy on Environmental Assessment was triggered with respect to subprojects under Component 1. However, the scale of the projects to be implemented under the workfare program, targeted at small community asset rehabilitation sub-projects, was such that no significant environmental impacts were expected.

b. Fiduciary Compliance:

SFD maintained satisfactory arrangements using their financial management system, and all Interim Financial Reports submitted by SFD were found to be acceptable. SWF also maintained adequate financial management arrangements. The audit reports for 2011 and 2012 were unqualified (clean).

The ICR reported that procurement activities were implemented efficiently and transparently.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Satisfactory
Satisfactory
 
Borrower Performance:
Satisfactory
Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

We agree with the lessons listed in the ICR, particularly the following two lessons:
Emergency operations embedded within the context of ongoing country strategies can be effective instruments. This project was provided through institutions that were at the center of the Bank’s assistance strategy in alleviating poverty, which facilitated the preparation of the Grant in an emergency manner, and ensured the lessons during implementation could be fed into the preparation of future operations in support of SFD and SWF.
The results framework and expected outcomes and outputs should be designed to reflect what an operation can reasonably accomplish and measure during implementation.

In addition, IEG offers two complementary lessons:
Depending on country context, particularly if an economic crisis emerges during project implementation, the number of participants in public works programs may be higher than expected, resulting in a lower benefit level and potential inclusion of non-targeted households/individuals.
Building up the financial management capacity of the implementing agency ahead of time is important to ensure effective project implementation.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is well-written and organized. It also provides a comprehensive economic and financial analysis for the project. However, there are some minor inconsistencies. For instance, the length of the workfare program was “nine to twelve months” per participating household on p. 5, then “spread over a period of six to nine months” on p. 6; or, "the average duration of employment offered to each household was approximately 50 days" in section 3.2 on p. 17, but 60 days in Annex 2 on p. 32.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions