Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Jordan Social Protection Enhancement Project


  
1. Project Data:   
ICR Review Date Posted:
06/06/2014   
Country:
Jordan
PROJ ID:
P100546
Appraisal
Actual
Project Name:
Jordan Social Protection Enhancement Project
Project Costs(US $M)
 10.88  2.99
L/C Number:
L7562
Loan/Credit (US $M)
 4.00  0.97
Sector Board:
Social Protection
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  06/03/2008
 
 
Closing Date
08/31/2013 08/31/2013
Sector(s):
Central government administration (75%), Other social services (25%)
Theme(s):
Social safety nets (67% - P) Administrative and civil service reform (33% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Pia Schneider
Judyth L. Twigg Lourdes N. Pagaran IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The objectives of the Project were to assist the Borrower in improving its management and delivery of the Cash Social Assistance Program aimed at reducing poverty among low-income households and individuals, as well as improving access to and quality of social work and care services for socially vulnerable families and individuals (Loan Agreement, p. 4).

    The Project Appraisal Document (PAD, p. 5) cites similarly worded project development objectives (PDO) but does not specify the target groups: to improve the management and operations of the cash social assistance programs and to improve the access to and quality of social care services.

    This review will use the project objectives as stated in the Loan Agreement.

    The PDO remained unchanged. However, in February 2013 during a Level-2 Restructuring, the target values for all three PDO indicators were changed (see Section 4).

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? No

Date of Board Approval:

c. Components:

The four components supported under the project are presented below, showing IBRD financing estimated at appraisal (including contingencies) and actual IBRD financing. The ICR does not report borrower contribution to each component, and therefore total project cost is not available by component.

      1. Institutional development and capacity building to formulate and implement social safety net programs (Appraisal: US$ 1.20 million from IBRD; Actual: US$ 0.396 million from IBRD). This component was to build capacity of institutions so that services would be delivered more efficiently and effectively. The three beneficiary agencies were the Coordination Commission for Social Solidarity (CCSS); the Ministry of Social Development (MOSD) to strengthen and develop its institutional capacity to analyze and manage social development and safety net policies and programs; and the Department of Statistics for human capacity strengthening.

      2. Renewal of the National Aid Fund (NAF) (Appraisal: US$ 1.10 million; Actual: US$ 0.08 million). This component was to improve the operation and management of the NAF; and improve coverage of the targeted poor population and targeting efficiency and effectiveness of the cash assistance paid to the population through the NAF. It was to implement the restructuring recommended for the NAF in the National Agenda. The component was to provide the NAF with a new targeting mechanism and adequate implementation procedures and arrangements, as well as the capacity to efficiently and effectively manage its operations and deliver targeted social assistance to poor households in Jordan. It was to establish an automated and up-to-date database on poor and vulnerable population groups in Jordan.

      3. Improving access to and quality of social work, care and rehabilitation services (Appraisal: US$ 1.40 million; Actual: US$ 0.325 million). This component was to improve access to and quality of services to vulnerable groups (including persons with disabilities, women and children in difficult circumstances, dysfunctional families, youth with behavioral problems, youth in conflict with the law, etc.). It would also include civil works and provision of materials, equipment, vehicles, technical assistance, and training. It was to develop standards of service and monitoring mechanisms; upgrade the model of care in three residential care institutions; pilot integrated social service centers; pilot resources and information for persons with disabilities; and pilot social innovation programs for the MOSD.

      4. Project Management (Appraisal: US$ 0.30 million; Actual: US$ 0.165 million). This component was to strengthen capacity of the MOSD, NAF, CCSS, and the Department of Statistics to efficiently and effectively implement the proposed project, while adhering to the Bank’s fiduciary requirements. Project activities were to be implemented by these agencies as part of their respective annual work plans. The MOSD was responsible for overall project implementation and was to establish a Project Implementation Support Unit (PISU), which was headed by the MOSD Secretary General. The PISU was to hire a project director (external consultant), a financial management specialist, and a procurement specialist. Fiduciary consultants were to handle the project’s fiduciary management in coordination with the MOSD for the first year, after which these functions would be absorbed by MOSD staff.

      Activities that were not progressing well under the four original components were cancelled during the restructuring in February 2013:

      1. MOSD activities were cancelled, including the formulation of a strategic development plan, human resource development, monitoring and evaluation activities, and development of a management information system (MIS). An advanced poverty analysis under the Department of Statistics was also cancelled.
      2. A NAF survey and new proxy means testing targeting system were cancelled as they were not approved by the Government;
      3. Two of three Integrated Social Service Centers, two of three MOSD resource information centers, and the piloting of a social innovation fund were cancelled.
      4. Financing was cancelled for a financial officer, project assistant, and secretary in the Project Implementation Support Unit.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost, Financing, Borrower Contribution

    • The actual project cost was US$ 3.01 million, compared to the original appraised amount of US$ 11.62 million net of taxes reported in the PAD, and the appraised project amount of US$ 10.88 million reported in the ICR. Actual costs were lower than appraised because of substantial delays in implementation and cancellation of activities during the February 2013 restructuring and at project closure.
    • The project was financed by a US$ 4.00 million IBRD loan, of which US$ 0.97 million actually disbursed. During the February 2013 restructuring, US$ 831,797 of the loan was cancelled, and the revised Loan amount was US$ 3.18 million. The total amount cancelled at project closure was US$ 3.003 million, as several project activities had not been implemented (see cancelled activities at restructuring above and in Section 4), and the Bank did not take appropriate action to address constraints to project implementation early on (see Section 8b).
    • The actual borrower contribution was US$ 2.02 million, compared to US$ 6.88 million planned at appraisal (29%). The counterpart funding requirement was estimated too high. Prior to 2009, Jordan reported strong economic growth, but the global economic crisis affected public finance and restricted the borrower contribution. In addition, frequent leadership changes affected government commitment (see Section 9a). During the February 2013 restructuring, the remaining borrower contribution was cancelled to avoid delays in procurement and payment processing (ICR, p. 12).

Dates
    • February 2013: Level-2 Restructuring. Activities were cancelled as described above, and related indicators and targets were adjusted. The borrower contribution for the remaining project period was cancelled, and the Bank became the sole financier of the project. The government agreed to strengthen financial management and procurement to expedite project implementation. The loan amount was reduced to $3.18 million (see above). The restructuring was to focus on activities that were progressing well, activities with high ownership and high probability of successful implementation. The closing date was not extended. Disbursement at this point was US$ 0.85 million (85% of the actual loan amount).


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Under the Original and Revised Targets: High
The project objectives of improved management of cash assistance programs and access and quality of social care are highly relevant. They were in line with Jordan's National Poverty Agenda (2006), which included the reform of social safety nets as a priority to reduce poverty (which was estimated at 13% in 2006). The objectives are also relevant to the current National Poverty Strategy 2013-2017, which aims to harmonize poverty reduction programs and improve targeting to the poor. The objectives are highly relevant to the Bank's Country Assistance Strategy FY12-15 goals of improved efficiency and effectiveness of public spending, and enhanced inclusion through social protection. Social safety nets reform and improved targeting are also highly relevant in the current context of declining economic growth and high unemployment, and the growing fiscal deficit that has restricted fiscal space for financing employment generation and social protection programs. The PDO also respond to the increased demand for expansion of social services, and to the tight fiscal situation that requires fiscal discipline. The government also aimed to strengthen social safety nets to prevent increased vulnerability as a result of the elimination of oil and fuel subsidies in 2008, and had approved several strategies to develop community-based social work and rehabilitation services tailored to the needs of vulnerable groups. Finally, the objectives are relevant in the context of the 2012 government announcement for a broadly targeted cash transfer to about 70% of the population to compensate for the fuel subsidy removal (ICR, p. 16).

b. Relevance of Design:


    Under the Original Targets: Modest
    Under the Revised Targets: Negligible

    The original project design incorporated relevant lessons from earlier operations and from the Bank's analytical work that needed to be addressed to achieve the project development objectives of improved management of the Social Assistance Program. Accordingly, the planned activities included institutional strengthening (component 1) to address an incoherent policy and institutional framework and overlapping institutional mandates and duplication; and the National Aid Fund (component 2) to improve ineffective targeting, discretionary decision making, weak monitoring and evaluation, and modest use of management information systems (MIS). The project's design (components and objectives) was aligned with the priorities identified in the government's National Poverty Agenda (2006). The originally planned activities were relevant to achieve project objectives and support the government in social protection reforms.

    The original results framework was weak and indicator targets too optimistic. These design weaknesses negatively affected project implementation.

    The revised results framework became unrealistic, as it was impossible to achieve the objectives with the remaining activities. Thus, the relevance of the revised design is rated negligible.


4. Achievement of Objectives (Efficacy) :


    PDO indicator targets and intermediate outcome indicator targets were scaled down or dropped during the February 2013 Level-2 restructuring to reflect the reduced scope of the project. There is no evidence to suggest that the project succeeded in contributing to achievement of the objectives, as only 25% of the original loan amount disbursed, most activities were cancelled, and those that had started were not completed (ICR, p. 18).

    Improving management and delivery of the Cash Social Assistance Program aimed at reducing poverty among low-income households and individuals,
    Under the Original Targets: Negligible
    Under the Revised Targets: Negligible

    Outputs
      • The MOSD did not adopt a Strategic Development Plan and a Human Resources Development Strategy. This indicator was dropped at restructuring and related project support cancelled.
      • The M&E Unit of the MOSD was not established and did not issue its annual report. The indicator was dropped at restructuring.
      • At least two-thirds of the MOSD staff did not complete training as envisaged in the Human Resources Development Strategy. No training was undertaken, as MOSD did not endorse the Strategy. The indicator was dropped at restructuring.
      • CCSS procured the standard packages for statistical data entry, cleaning, imputation, management, and analysis, and staff were trained in using them. However, CCSS was abolished in 2010 during a public expenditure management consolidation. The project financed the procurement of STATA and SPSS software and financed training of staff at the Department of Statistics in software use.
      • NAF did not adopt an Operations Manual to govern its business processes and performance.
      • No training was held for NAF HQ and MOSD/NAF local branches. No information is provided on improved staff skills.
      • No information is provided on the use of smart cards to collect benefits. The indicator was dropped at restructuring.

    Outcome
      • Coverage of the targeted poor population by NAF programs increased from 19% in 2008 to 25% in 2013, not meeting the original target of 70% and the revised target of 60%.
      • The error of inclusion of the non-poor was not available at project closure. It was 80% at baseline against an original and revised target of 30%.
      • No information is provided on the increase in the number of people benefiting from the MOSD social work and care services in localities served by pilot centers. The original target was a 50% increase. The revised target of "100% of pilot centers fully functioning" is not linked to the indicator.


    Improving access to and quality of social work and care services for socially vulnerable families and individuals.
    Under the Original Targets: Negligible
    Under the Revised Targets: Negligible

    Output
      • Standards of care were developed but not adopted and not implemented, not meeting the original and revised target of 60% of standards of care adopted and staff trained to implement them.
      • The MOSD transformed all residential institutions to resemble a family-like setting, outside the project. The project financed some furniture, computers, and electronics, not meeting the target of financing the transformation of 3 (original target) and 1 (revised target) institutions.
      • Three pilot integrated social work and care centers were fully functional (Amman, Aqaba, and Irbid). The project's contribution to these centers is unclear.
      • One center was rehabilitated for persons with disability, not meeting the target of 3. However, the construction financed under the project was not compliant with building code standards, and the center was subsequently abandoned in 2013.

    Outcome
      • User satisfaction with the NAF and MoSD services was not measured. The original target was "satisfactory," but not defined. The revised target was "not available."
      • There was no indicator to measure improved access to social care for vulnerable individuals.
      • There was no indicator to assess improved quality of social work and care delivered to vulnerable individuals.


5. Efficiency:


    Negligible.

    The PAD and ICR did not provide a cost-effectiveness analysis. At appraisal, it was expected that improved targeting of NAF assistance to the poor through proxy-means testing would increase the share of the poor reached from 20% to 81%. In addition, improved quality of social care services to vulnerable population was expected to lead to higher individual and overall productivity; as well as decreased social costs and associated public spending. However, the proxy means testing was not implemented and cancelled during the 2013 restructuring. The expected benefits have not materialized.

    There were severe inefficiencies in the use of project resources. More than half of project funds were spent on project management. However, there were substantial delays in project implementation caused by weak fiduciary management, insufficient commitment to the reforms to be supported by the project, and inadequate management oversight to address these weaknesses. The loan financed only a few disparate activities, and 75% of the loan was cancelled.



a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


    Outcome under original PDO targets: Highly Unsatisfactory.
    Relevance of the objectives was High, while relevance of the original design was Modest. Achievement of the objective of improved management of the cash social assistance program was Negligible, and the achievement of improved access and quality of social work was also Negligible. Efficiency is rated Negligible. These are severe shortcomings in the operation's preparation and implementation, resulting in an Outcome rating of Highly Unsatisfactory.

    Project under revised PDO targets: Highly Unsatisfactory.
    Relevance of the objective was High, while relevance of the revised design was Negligible. Achievement of the objective of improved management of the cash social assistance program was Negligible, and the achievement of improved access and quality of social work was also Negligible. Efficiency is rated Negligible. These are severe shortcomings in the operation's preparation and implementation, resulting in an Outcome rating of Highly Unsatisfactory.

    According to OPCS/IEG guidelines for restructured projects, the final outcome rating is determined according to the percentage of the credit that disbursed before and after project restructuring. This IBRD loan had disbursed $0.85 million (85% of actual loan amount) at the time of restructuring. The ratings under the original and revised PDO targets are both Highly Unsatisfactory. Therefore, the combined outcome rating is Highly Unsatisfactory.


a. Outcome Rating: Highly Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:


    The risk to development outcome is high, as project objectives have not been achieved. A new project financed by the Deauville Partnership Transition Trust Fund (US$ 9.5 million) and managed by the Bank aims to achieve a similar set of objectives but through different means. The project is supporting the implementation of a national unified registry and outreach program and is expected to improve the accuracy of targeting of low-income households and strengthen targeting systems used by the NAF.


    a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:

The Bank team prepared an operation that was responsive to the government's reform needs and the Bank's country strategy. However, the project was too ambitious. The Bank team overestimated local implementation capacity, given that this was the first externally financed project of this scale to be implemented by the MOSD. The Bank identified the relevant risk factors during preparation, but it underestimated the risks of “weak project implementation capacity” and “increased regional political instability that might shift Government focus and slow down reform implementation," which both materialized. The results framework was inadequate.

During preparation, the design did not accurately incorporate the views of the various agencies, as a result of which the government was not prepared for this type of project. The design also overestimated sustained government commitment to reforms and the need for policy decisions which the government found difficult to implement. The institutional arrangements to implement the project design were too optimistic given the capacity constraints. Implementation involved several agencies that were not well coordinated. There was no higher-level steering committee to guide reforms. And there was a disconnect in the implementation approach. The design empowered the project implementation support unit (PSIU) to designing reforms and supporting the government in the implementation of these reforms.

Quality-at-Entry Rating: Unsatisfactory

b. Quality of supervision:


    During five years of implementation, there were four Bank task team leaders. Although project supervision identified weaknesses in implementation, the Bank team did not take appropriate action. The mid-term review was delayed to February 2012. The aide-memoire of the Mid-Term Review was not finalized, and the few supervision reports mainly focused on the second component. During the mid-term review the Bank team discussed restructuring, but it took one year for the Bank to finalize this restructuring. The Bank did not recognize the political economy situation, including a changing view of the government on the NAF and proxy means testing and how to treat beneficiaries of the cash program who would not receive benefits anymore. The Bank team did not respond to the government's changing needs, priorities, and institutional realities (ICR, p. 17).

    The Bank had a strong focus on the proxy means testing for targeted assistance, which created a disconnect with government priorities. The Bank also did not adequately inform stakeholders about the benefits and cost of the new system to create commitment, and did not support communication among project components and participants. These issues could have been addressed during an early project restructuring. Project implementation was rated Moderately Unsatisfactory from 2011, but the Bank only became proactive six months before project closure and restructured the project. During the 2013 restructuring, several activities were cancelled that did not have government support. The project development objectives were not changed to reflect the reduced project scope. The resulting changes were inadequate to improve implementation performance. The Bank kept the project to maintain the policy dialogue, but this dialogue was not effective. These issues could have been resolved through an early right-sizing of the project with a Level-1 restructuring.

Quality of Supervision Rating: Highly Unsatisfactory

Overall Bank Performance Rating: Highly Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

During project preparation, the government showed ownership and supported the planned reforms. However, during implementation there were frequent changes in leadership (five ministerial changes from 2007 until 2013) and changes in the director of NAF, as well as staffing changes, which all negatively affected ownership and interest for the project. There was passive commitment to the renewal of NAF, which was supported under the second component. In 2010, the Jordanian cabinet of ministers abolished the Coordination Commission for Social Solidarity (CCSS), which was responsible for social research, and transferred its mandate to the MOSD. At the restructuring in 2012, the government did not request an extension of the closing date to implement the remaining activities. However, it did agree to put in place enhancements in financial management and procurement to expedite procurement. The government's counterpart contribution was 29%, considerably less than the originally planned amount.

Government Performance Rating: Unsatisfactory

b. Implementing Agency Performance:

The PISU was not in a position to coordinate the five agencies involved in the implementation of project activities. The unit was without a director for intermittent periods, including during the first 9 months, which negatively impacted implementation. During the entire project period, the PISU lacked capacity to implement its functions, including in fiduciary management. Fiduciary management was weak, which contributed to implementation delays (see Section 11). The PISU did not develop a communication plan to better coordinate and inform about the project. Project M&E was inadequate (see Section 10).

Implementing Agency Performance Rating: Unsatisfactory

Overall Borrower Performance Rating: Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

Analytical work was available at appraisal; however, the M&E design did not provide relevant baseline data. Several relevant indicators were missing. The NAF was to rely on the Household Income and Expenditure Survey (HIES) to identify coverage (through benefit incidence analysis), targeting efficiency, and poverty impact, as well as a special survey of NAF beneficiaries that was never carried out. The target values for improved coverage were overly optimistic, including that NAF coverage would increase from 19 to 70% of the poor without putting in place intermediate indicators to deal with outreach and enrollment for the poor who were not covered. Also, it was not realistic to expect that NAF leakage would decrease without any graduation schemes or services for beneficiaries who did not qualify for benefits.

b. M&E Implementation:

The Institutional Development and Capacity Building Component was to set up an M&E unit to coordinate M&E across programs, but no payments were made for these activities under the project and the MoSD did not approve the recruitment of an M&E consultant. M&E activities were to be managed by the PISU and results disseminated to the public (ICR, p. 14). Progress on most project indicators was not measured (see Section 4). The PISU compiled information on components by using information submitted at disbursement processing and conducted site visits to monitor intermediate outcome indicators. M&E implementation relied mostly on outsourced M&E by a third party. Neither the Ministry of Social Development nor NAF developed M&E capacity, even though the project was intended to strengthen the government’s capacity in such areas. Baseline information was not collected when the loan became effective.

a. M&E Utilization:

M&E data and analysis were not used to guide or adjust project implementation. The department of statistics routinely carried out household surveys. Its capacity for household data collection and poverty measurement and monitoring was strengthened under the Project.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

This is a category C operation. No safeguard policies were triggered.

b. Fiduciary Compliance:
Financial management and procurement were weak, which led to implementation delays. Frequent staff changes created a knowledge gap about procedures and led to insufficient coordination between financial management and procurement (ICR, p. 15). Fiduciary performance was rated moderately unsatisfactory during most of the project period.


Financial management was rated Unsatisfactory in most supervision reports due to delays, including for loan withdrawal applications. However, the PISU maintained an acceptable automated accounting system and adequate internal controls. All audit reports until the end of 2012 were unqualified but submitted with substantial delays. The December 2012 auditor's opinion was qualified, and a disclaimer of audit opinion was issued on the statement of fixed assets (ICR, p. 15). Ineligible expenditures were identified at project closing (taxes financed from Bank loan) and had to be reimbursed by the MOSD.

Procurement: Procurement was weak due to staffing changes, delays in processing, and non-adherence to procurement rules (ICR, p. 15).

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Highly Unsatisfactory
Highly Unsatisfactory
 
Risk to Development Outcome:
High
High
 
Bank Performance:
Unsatisfactory
Highly Unsatisfactory
The Bank was not responsive to a changing political economy context, and it was not proactive in restructuring the project in a timely manner. 
Borrower Performance:
Unsatisfactory
Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The ICR (pp. 26-28) identifies several valuable lessons:

Problem projects require intensive implementation support to become successful. Restructuring can be used at the beginning of a project to adjust its scope to changing contexts. In this case, the restructuring was done too late and was inadequate to affect project performance.

Poorly performing projects should not be maintained just to sustain policy dialogue. In this case, the project was not closed, in part because it was seen as providing a vehicle for policy dialogue (ICR, p. 27). However, this dialogue appeared to be ineffective, and the government has to pay a lending rate for this fixed spread loan.

Projects need to be kept flexible. In many countries, a combination of targeting mechanisms are used to reach the poor. In this case, the Bank's focus on proxy-means tested targeting was not flexible and did not respond to the government's concerns with this method. The Bank could have used an opportunity for consensus building early on and adjusted the targeting methods to the local context.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR is very well written and contains good analysis. It is candid in its assessment and appropriately critical. Information is well supported with sources. The ICR is internally consistent and consistent with guidelines. It also presents very insightful and practical lessons and discussion of project implementation. The ICR does not conduct a split rating analysis of project performance before and after PDO targets were changed.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions