Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Gz-social Safety Net Reform Project


  
1. Project Data:   
ICR Review Date Posted:
06/24/2013   
Country:
West Bank & Gaza
PROJ ID:
P081477
Appraisal
Actual
Project Name:
Gz-social Safety Net Reform Project
Project Costs(US $M)
 82.5  28.4
L/C Number:
Loan/Credit (US $M)
 10.0  28.4
Sector Board:
Social Protection
Cofinancing (US $M)
 0  0
Cofinanciers:
Board Approval Date
  06/03/2004
 
 
Closing Date
12/31/2008 06/30/2012
Sector(s):
Other social services (90%), Central government administration (10%)
Theme(s):
Social risk mitigation (29% - P) Social safety nets (29% - P) Education for all (14% - S) Child health (14% - S) Other public sector governance (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Judith Hahn Gaubatz
Judyth L. Twigg Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Project Appraisal Document (PAD, page 7) and the Trust Fund Grant Agreement (Schedule 2), the original project objectives were:
      • To protect and enhance the human capital of children from the poorest households in the West Bank and Gaza; and
      • To strengthen the institutional capacity of the Ministry of Social Affairs needed to assist in efficiently and effectively carrying out the Special Hardship Case social assistance program.

    Due to the deteriorating fiscal situation of the borrower, along with external donors' temporary suspension of funds, the expected leveraged funds (over US$70.0 million) did not materialize. Therefore, the project scope was significantly reduced. The project objectives were revised as follows (Project Paper, page 2):
      • To mitigate the impact of the continued socio-economic crisis on a subset of the poorest and most vulnerable households; and
      • To strengthen the Ministry of Social Affairs' institutional capacity to manage cash transfer programs.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 06/19/2007

c. Components:
Component 1: Special Hardship Case (SHC) Program Reform (Appraisal: US$80.0 million; Actual: US$26.0 million): This component aimed to finance the expansion of the existing Special Hardship Case social assistance program by adding a conditional cash transfer component. The new component aimed to support investment by very poor and vulnerable households in the health, nutrition, and education of their children. The cash transfer component was expected to reach 30,000 households, each of whom would meet eligibility criteria based on a proxy means test. The cash transfer payments would be comprised of a health grant (for food, micronutrients, and health expenditures) to keep preschool children nourished and healthy; an education grant (for out-of-pocket school expenditures such as school supplies, uniforms, and transportation); and an awareness grant to encourage participants to attend sessions on health and education. Beneficiaries were required to meet conditions such as quarterly health clinic visits, regular school attendance, and participation in counseling sessions.

Component 2: Capacity Building (Appraisal: US$1.6 million; Actual: US$2.36 million (combined with Component 3 actual amount)): This component aimed to strengthen the institutional capacity of the government agencies implementing the project, including the Ministry of Social Affairs (MOSA), the Ministry of Health, the Ministry of Finance, and the Ministry of Education. Activities included: technical assistance on policy and planning; strengthening of the MOSA's management information system; provision of training and equipment; and technical assistance to conduct a public awareness campaign.

Component 3: Project Management (Appraisal: US$0.9 million; Actual: US$2.36 million (combined with Component 2 actual amount)): This component financed project management costs and monitoring and evaluation activities.

Revisions to project components:

    • Under the formal project restructuring (June 2007), the name of Component 1 was changed to "Pilot Cash Transfers to Poor Households - provision of cash grants to support poor households selected using an appropriate targeting system." The focus was shifted to a pilot approach, the scope of participation was reduced to 4,000-5,000 households, and beneficiaries were no longer required to meet the conditions related to education and health. According to the Project Paper (page 3), the pilot approach would test whether a sustainable cash transfer program could be established (assuming a rigorous targeting mechanism) that could be scaled up during periods of economic downturn.
    • Under the first Additional Financing, provided through the Trust Fund for Gaza and the West Bank, additional support was provided to Component 1 to expand the number of beneficiary households by 2,000.
    • Under the second and third Additional Financing, provided through the Food Price Crisis Response Trust Fund, additional support was provided to Component 1 to finance one-time grants (the first grant was in the amount of US$200, the second US$135) to approximately 25,000 households, in order to offset global increases in food prices.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost

    • According to the PAD (page 1), the appraised project cost was US$82.5 million, of which the borrower was expected to provide US$57.5 million. The remaining financing gap of US$15.0 million was expected to be met by other donors.
    • The grant had originally intended to leverage significant funding from the borrower and from other donors; however, this funding did not materialize due to external circumstances, namely increased political instability in the territory, fiscal crises, and the suspension of donor aid. Under the formal project restructuring, the project scope was significantly reduced, and the estimated project cost was revised to US$10.0 million.
    • Through three subsequent Additional Financings, the project amount was then increased to a total of US$28.4 million.

Financing
    • The project was initially financed by a trust fund grant from the Bank-administered Trust Fund for Gaza and West Bank, amounting to US$10.0 million total. One Additional Financing request was approved, also from the Trust Fund for Gaza and West Bank, for US$10.0 million. Second and third Additional Financing requests, in the amounts of US$5.0 million and US$3.4 million, were approved, both from the Food Price Crisis Response Trust Fund.
    • In July 2005, the grant agreement was amended to reflect new country financing parameters, in which 100% of expenditures were now eligible for Bank-administered trust fund financing.
    • In June 2006, the grant agreement was amended to allow funding of unconditional cash transfers, instead of only conditional cash transfers.

Borrower contribution
    • The borrower was expected to provide US$57.5 million in counterpart financing. However, due to the deteriorating fiscal situation of the ruling Palestinian Authority, the borrower obligation could not be met and the actual contribution was $0.

Dates
    • In June 2007, the project was formally restructured so that the project development objectives, associated outcome indicators, and targets were revised. US$2.1 million, or 7.65%, of the project grants had disbursed at the time of the restructuring.
    • In March 2008, Additional Financing from the Bank's Trust Fund for Gaza and West Bank was approved in the amount of US$10.0 million. The funds were to be used solely for Component 1, in order to scale up the cash transfer pilot program into a government program. The Additional Financing was also expect to leverage funds from other donors such as the European Union. The project closing date was also extended from December 2008 to June 2012.
    • In October 2008, a second Additional Financing from the Bank's Food Price Crisis Response Trust Fund was approved in the amount of US$5.0 million.
    • In March 2010, a third Additional Financing from the Bank's Food Price Crisis Response Trust Fund was approved in the amount of US$3.4 million.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Original and Revised: High
Due to the continuing insecurity and political instability in the territory, economic and social indicators have deteriorated significantly. The poor are facing increasingly difficult circumstances in which consumption of basic food and non-food essentials is decreasing, with 16% of the poor unable to afford the cost of basic food and non-food subsistence. Recent studies show that malnutrition among children is on the rise, and school has become less affordable for families, putting school attendance rates at risk. The country's National Development Plan for 2011-2013, as well as the Bank's Interim Strategy Note for 2012-2014, identify strengthening of public institutions and providing social assistance, specifically targeted to those most in need, as main priorities.

b. Relevance of Design:

Original: Modest
A Bank study in 2003 showed that transfer instruments were having a greater welfare impact in the area than other welfare instruments such as job creation schemes. Thus, direct cash transfer payments to the most vulnerable households were likely to contribute to the intended outcome of sustaining the well-being of children in those households. The explicit targeting element of the program (utilizing a proxy means test formula based on 31 variables) ensured that the poorest households would be reached. However, the conditional aspect of the program (requirements for health clinic visits and school attendance) was less likely to contribute to the intended outcomes, due to the lack of enforceability of such conditions.

Revised: Substantial
The restructured project was significantly reduced in scope, in terms of the number of intended beneficiary households. It also shifted to a pilot approach for testing the cash transfer program. By strengthening the institutional capacity of the Ministry of Social Affairs, particularly in implementing the poverty targeting mechanism, the program could be scaled up in the continued socio-economic crisis situation. The original conditionalities related to health and education were removed.


4. Achievement of Objectives (Efficacy) :


Note: By the project closing, the social assistance programs had been merged into one unified cash transfer program, which was funded by the Bank, the European Union (EU), and the Palestinian Authority (PA). During the period 2006-2010, the Bank provided US$26.0 million, or 11%, of the total financing; the EU provided 85%; and the PA provided 4%.

To protect and enhance the human capital of children from the poorest households in the West Bank and Gaza (Original)
Negligible, due to significant shortfalls in implementation and results.

Outputs
    • Fewer than 2,000 households, out of the targeted 30,000, had received cash transfer payments. There were long delays in disbursing cash payments, with the first round of payments delivered two years into the project period. The ICR (page 13) reports that beneficiaries characterized the payments as inconsistent and difficult to access (due to the closing of some local post offices where payments were to be picked up).
    • Conditionalities related to health and education were not monitored because of severe restrictions on movement within the territory, and the lack of formal arrangements among the relevant Ministries (Health and Education) to verify compliance.

Outcomes
None reported.


To mitigate the impact of the continued socio-economic crisis on a subset of the poorest and most vulnerable households (Revised)
Substantial, due to effectiveness of targeting the poorest households, and some evidence of mitigated impact on the project households.

Outputs
    • 4,500 to 5,000 households were receiving regular payments during the period August 2007 -December 2010 (project period under the original $10.0 million grant and the first AF of US$10.0 million), with the average payment ranging from US$228 -$347. The payments disbursed were based on the amount needed to bridge 50% of the poverty gap in each household.
    • About 25,000 households received two payments (US$200 and US$135) during the period June 2009 to December 2010 (project period under the second and third AFs of US$5.0 million and US$3.4 million).
    • Various assessments indicate that the poorest households were indeed being reached by the program. The 2009 Eligibility Review concluded that although there was some leakage and under-coverage, the beneficiary households did meet the stated eligibility requirements. The 2010 Impact Evaluation of the second AF showed evidence that beneficiaries of the program were poorer than non-beneficiaries. The 2012 Targeting Assessment for the Bank's follow up Cash Transfer Project (which utilizes the same poverty targeting database and reaches the same population) indicated that among program applicants, 70% of those classified by the poverty formula as "extremely poor" were in fact among the poorest 10% of the population.
    Outcomes
      • Findings from the 2010 Impact Evaluation of the second AF included the following: 59.3% of beneficiaries reported that their food consumption improved (referring to both quality and quantity) after receiving the one-time food grant, compared to the period before having received the grant.
      • There has been a four percent reduction in poverty in the territory since 2004. Given that the project was effective in reaching the poorest households and that cash payments were calculated to bridge 50% of the poverty gap, the overall reduction in poverty can plausibly be attributed in part to the project's support.


    To strengthen the institutional capacity of the Ministry of Social Affairs to carry out the Special Hardship Case social assistance program/ to manage the cash transfer program (Original and Revised)
    Substantial, due to evidence of effectiveness of the management information system and the scalability of the social assistance program.

    Outputs
      • Ministry of Social Affairs (MOSA) staff at central and district levels received training in project management, use of information technology, and fiduciary responsibilities. MOSA social workers received training on conducting of beneficiary interviews, survey data collection, data entry, and support services for beneficiary households.
      • Equipment and technical assistance were provided to upgrade the management information system. The upgraded system is a web-based network that links district offices to the central office database and has the capacity to track household data, calculate household benefits levels, provide a poverty ranking of all beneficiary households, track applicant status, and produce M&E reports. The system is used to ensure that the poorest households are targeted for the cash transfer payments.

    Outcomes
      • According to an assessment of the follow-up Cash Transfer Project (which utilizes the same poverty database and reaches the same population), 84% of beneficiaries in the West Bank and 70% of beneficiaries in Gaza expressed satisfaction with the program in terms of the approach, application process, and timeliness of payments. Most respondents did, however, indicate that they would prefer higher levels of support.
      • According to the 2009 Eligibility Review in which 10% of beneficiary households were assessed, 90% were confirmed as "eligible" according to the poverty criteria. At the start of the project period, only 48% of beneficiaries of the social assistance program were confirmed as poor. The targeting system is also being used by other social assistance programs for targeting and designing their own programs.
      • The scalability of the program was effectively demonstrated through the implementation of the second and third AFs, which provided one-time grants from the Food Price Crisis Response program. It has also been reflected by the scaling up of the overall cash transfer program to now serve 95,000 households.

5. Efficiency:

Original: Modest
Revised: Substantial

The project design used a targeted approach (to identify the poorest households) as well as flexible benefit levels per household (based on the household's specific poverty level), rather than a blanket amount to all households. These project elements likely contributed to the efficient use of project resources, although for the project under the original objectives, these elements were implemented only to a limited extent.

Following the project restructuring, significant funds were leveraged from other donors to multiply the project's impact. The single unified targeting database has led to administrative efficiencies in the Ministry of Social Affairs, as well as reduce the risk of double payments. The latter outcomes related to institutional capacity were achieved at 8% of the total project cost.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Original objectives
Unsatisfactory. The relevance of the project objectives is rated High, while the relevance of the project design is rated Modest due to the lack of enforceability of cash transfer conditions. Achievement of the objective to protect and enhance the human capital of children from the poorest households is rated Negligible due to the significant shortfall in reaching the intended number of beneficiary households. Achievement of the objective to strengthen the institutional capacity of the Ministry of Social Affairs is rated Substantial. Efficiency is rated Modest

Revised objectives
Satisfactory. The relevance of the project objectives is rated High, while the relevance of the project design is rated Substantial. Achievement of the objective to mitigate the impact of the socio-economic crisis on the poorest and most vulnerable households is rated Substantial, due to effectiveness in targeting the poorest households, and some evidence of mitigated impact on the project households. Achievement of the objective to strengthen the institutional capacity of the Ministry of Social Affairs is rated Substantial, due to evidence of effectiveness of the management information system and the scalability of the social assistance program. Efficiency is rated Substantial.

Overall project outcome
Satisfactory. According to the harmonized OCPS/IEG guidelines on rating restructured projects, the overall outcome rating is derived by weighting the outcome rating for the original project and the revised project, according to the amount of the grant disbursed at the the time of restructuring. In the case of this project, the grant had disbursed US$2.17 million, or 7.65%, of the total grant at the time of restructuring. Therefore, the overall outcome rating is Satisfactory (4.77, which rounds to 5.0 out of 6.0 on the six-point rating scale).

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:


Institutional capacity in MOSA is likely to be sustained. A follow-up Bank project (Cash Transfer Project, approved in May 2011 for US$10.0 million) will support the continuation of the reform agenda, as well as direct cash payments to beneficiary households. However, longer-term financial sustainability is highly uncertain given the heavy reliance on donor funds and the fiscal and political instability in the territory. This, coupled with a likely increase in the number of eligible households, puts the mitigation outcomes at high risk.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The project design drew substantially from findings of Bank analytic work in the territory (2002 Social Safety Net Report, 2002 report on social assistance programs) as well as lessons from conditional cash transfer programs in other countries (including the usefulness of similar mechanisms to move from welfare to behavior change, and the critical role of monitoring and evaluation based on an effective management information system). However, the capacity of the government agencies to enforce the conditionalities was overestimated, including the substantial coordination needed to monitor compliance. Overall risk was appropriately assessed as High, including the risks of political instability, delays in co-financing or counterpart funding, and weak capacity of participating ministries. Although these risks did materialize, the project was restructured and thereby rendered some of these risks less relevant. The M&E framework had shortcomings, including the lack of baseline data or precise target values, and there was inadequate capacity to monitor outcomes related to the health and education conditionalities.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

Implementation during the first two years of the project period was slow, with the first cash grant payments disbursed to beneficiaries two years after project effectiveness. During a 2006 Country Portfolio Performance Review conducted by the Bank, agreement was reached to restructure the project in light of the changed socio-economic and political circumstances. Along with the reduced target number of beneficiaries, critical adjustments were made to the project design, such as removing conditionalities and shifting payment modalities to commercial banks instead of post offices. Proactive supervision, marked by responsiveness to the situation on the ground, was enabled by field-based presence of Bank staff and continuity in the task team members. The M&E framework was revised to reflect the reduced scope of the project. Procurement performance was overall satisfactory, and initial shortcomings in financial management were addressed through additional training. There were no safeguard policies triggered by the project.

Quality of Supervision Rating: Highly Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

During the initial project period, there was significant political unrest and security concerns that affected implementation on the ground. For example, the Ministry of Social Affairs (MOSA) office was moved from Gaza to the West Bank, leading to disruptions in implementation, including the need to train new staff. Escalating military conflict in Gaza also led to severe restrictions on movement in the territory (causing difficulties for MOSA staff in reaching beneficiary households) and various civil servant strikes. A fiscal crisis led to the inability of the borrower to fulfill its counterpart financing commitment. However, a number of these concerns (including a temporary suspension of donor aid) were due to external circumstances that could not be anticipated and that were beyond the control of the government. In the midst of these challenges, the PA remained committed to the overall social assistance reform agenda and to project implementation after the restructuring. In particular, the PA strongly supported the use of the poverty targeting mechanisms, took advantage of the Additional Financings to scale up the project's reach to more households, and eventually merged the Bank project with the EU project into one nationally-administered program.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:

During the initial project implementation period, the Ministry of Social Affairs (MOSA), which was the implementing agency primarily responsible for disbursing cash grants and monitoring adherence to conditions, faced significant difficulties due to the ongoing political and socioeconomic crisis. There was a two-year delay in the disbursement of cash transfer payments. However, MOSA was overall effective in disbursing all payments as planned and in implementing the critical management information system. The project management unit provided effective monitoring of the project, including regular reporting on project progress. Procurement performance was overall satisfactory, with some initial shortcomings in financial management. There were no safeguard policies triggered by the project.

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The original M&E framework had shortcomings, including the lack of baseline data (which was to be collected during a pilot targeting phase) or precise target values (a number of which were identified as ranges such as 2%-15% increases). The original indicators for health and education outcomes were unlikely to accurately capture the project's impact; the Project Paper (page 2) notes that health and education indicators remained high despite ongoing socio-economic difficulties; therefore the marginal impact of the project on these indicators was likely to be insignificant. Evaluative activities were included in the M&E design, such as a beneficiary assessment, study on the effectiveness of the targeting mechanisms, and audits to verify data quality.

b. M&E Implementation:

The management information system was made fully operational and was critical to implementation of the cash transfer program, particularly in ensuring that the targeting mechanism was effective. Planned evaluative activities were carried out, although some were conducted for the follow-up Cash Transfer Program.

a. M&E Utilization:

The data collected and monitored in the management information system was utilized to monitor the cash transfer program, to determine eligible households, and to set beneficiary payment levels. The system is now being used in the unified cash transfer program rolled out nationwide. The information was also used to inform decision-making related to planning and resource allocation for the social protection sector.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

The project was classified as an Environmental Category "C" project; no safeguard policies were triggered.

b. Fiduciary Compliance:

Procurement: Multiple small packages were procured during the initial years of the project period using simplified procurement procedures. No major problems were reported. However, procurement performance diminished following the resignation of the procurement specialist in the project management unit (the ICR does not state when this resignation took place). One case of misprocurement was reported regarding the purchase of 50 computers; subsequently, US$43,166 was cancelled from the trust fund grant and the amount was eventually refunded to the special account. Measures were taken to mitigate misprocurement risk, including the requirement to subject all procurement packages to prior review by the Bank, and the hiring of a part-time procurement consultant in the project management unit. Overall, procurement performance was satisfactory.

Financial management: The project management unit experienced some initial difficulties in financial management, including a few instances of non-compliance with Bank fiduciary guidelines and the trust fund grant agreement. However, financial management performance improved by 2008, through additional training for project staff, technical support from the Ministry of Finance, and a completed financial management manual. Audit reports were overall timely, with one qualified report. The ICR does not provide details on the qualification, nor whether the issue was resolved.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Satisfactory
 
Risk to Development Outcome:
High
Significant
Institutional capacity outcomes are likely to be retained, although the sustainability of improved livelihoods for the poorest households is highly uncertain. 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Satisfactory
Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The following lessons are adapted from the ICR (pages 30-32):
    • Conditional cash transfers are not the best approach in all contexts. In the case of the West Bank and Gaza, parents were already motivated to send their children to school and to health clinics. However, families needed cash transfers to assist in these activities. Coupled with the complexity of enforcing the conditionalities in this particular context, a conditional cash transfer approach was not the best fit.
    • A poverty means-testing mechanism can be an effective modality for targeting beneficiaries. In the case of this project, the proxy means test was objective, non-politicized, and largely accurate in identifying the poorest households. It was also useful for a quick scaling up of the program during crisis situations.
    • Structural reforms of a social safety net program may need to be prerequisities for establishing a cash transfer program, particularly in a context of limited institutional capacity. In the case of this project, ensuring the use of an effective and transparent targeting mechanism, and using the banking system to transfer cash payments, made significant contributions to the project's achievements.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is notable for its clarity, particularly in describing the evolution of the project and discussing the results chain. The quality of the evidence was adequate, although stronger evidence of the impact of cash transfers on the livelihoods of beneficiaries would have further substantiated the project's achievements. Lessons are well drawn from the project experience.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions