Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Social Assistance System Modernization Project


  
1. Project Data:   
ICR Review Date Posted:
11/11/2013   
Country:
Ukraine
PROJ ID:
P075231
Appraisal
Actual
Project Name:
Social Assistance System Modernization Project
Project Costs(US $M)
 101.78  106.20
L/C Number:
L4807
Loan/Credit (US $M)
 99.40  92.56
Sector Board:
Social Protection
Cofinancing (US $M)
   3.93
Cofinanciers:
SIDA
Board Approval Date
  11/17/2005
 
 
Closing Date
12/31/2008 01/01/2013
Sector(s):
Other social services (70%), Central government administration (15%), Sub-national government administration (15%)
Theme(s):
Social safety nets (50% - P) Social analysis and monitoring (25% - S) Other social protection and risk management (25% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Hjalte S. A. Sederlof
Judyth L. Twigg Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:

The Project Development Objective (PDO) as set out in Schedule 2 of the Loan Agreement was “to assist the Borrower in achieving significant improvement of the effectiveness of Ukraine’s social assistance system by better targeting of the cash benefits and the reduction of burden on beneficiaries.

The objective statement had two sub-objectives: “better targeting of cash benefits” and “reduction of burden on beneficiaries.”

The PDO in the Project Appraisal Document was to seek "significant improvement of the effectiveness of Ukraine's social assistance."

Achievement of the PDO will be assessed against the sub-objectives in the PDO for the Loan Agreement..

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

Component 1. Administrative Modernization (estimated cost at appraisal US$ 84.21 million; actual cost US$ 87.09 million). The component was to include modernization and improvement of the capacity of the local and regional social welfare offices to enable delivery of better quality services to clients. It was to support organizational and operational reforms, including the creation of a unified, function-based and client-oriented service; and development of capacities for effective implementation of the ongoing social assistance reform program. It was to finance activities at the central level to be implemented by the Ministry of Labor and Social Protection (MOLSP), and to provide subsidies at the oblast (regional) level for modernization of local social welfare offices.

Component 2. Social Protection Policy Development, Monitoring, Evaluation and Public Information (estimated cost at appraisal US$ 4.40 million; actual cost US$ 3.66 million). The component aimed at building MOLSP capacity for developing social assistance policies, with a systemic approach to monitoring program outcomes and policy impacts. It also was to develop capacity of local and regional social welfare offices for outreach to the poorest parts of the population, to try to include them into the social assistance programs financed by the Government.

Component 3. Institutional Development (estimated cost at appraisal US$ 0.62 million; actual cost US$ 1.34 million). This component was to strengthen institutional capacity of the MOLSP for efficient and effective implementation of the program.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project cost and financing. Project cost at appraisal was estimated at US$ 101.78 million. The actual project cost was US$ 106.2 million. Financing was provided by IBRD (US$ 99.4 million) and the Swedish International Development Agency. The SIDA contribution was denominated in Swedish kronor, SEK. It amounted to SEK 29.4 million (approximately 3.5 million at the signing of the agreement). It was to provide technical assistance for development of capacity in the MOLSP, the Ministry of Finance, and the State Statistical Committee of Ukraine for: (i) impact evaluation of social prot4ection programs, including methodological guidelines; (ii) results-based management and budgeting; and (iii) public education and communication of ongoing social assistance reforms and policies.

Borrower contribution. The Borrower contributed US$ 4.94 million of a planned US$ 2.38 million. The ICR does not explain why the Borrower contribution was higher than anticipated.

Dates. The project was approved on November 17, 2005. Its original Closing Date was December 31, 2008. It was extended four times to its final Closing Date of January 1, 2013. The Project was extended three times to accommodate delays in implementation and changes in the Ministry's top management. The first extension (from December 31, 2008 to May 30, 2010) was granted to correct misjudgment in implementation capacity. The second extension (from May 30, 2010 to July 1, 2011) was granted to complete civil works and activities related to the management information system (MIS). The final extension (from July 1, 2011 to January 1, 2013) was granted to complete the MIS rollout and to further institutional and capacity building that could feed into the preparation of a subsequent project.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Relevance of objectives is rated modest.

At the time of project preparation, Ukraine was spending 1.2 percent of GDP on a social assistance system that included numerous, often overlapping, ineffectively targeted programs, weak administration, and inadequate monitoring and evaluation. The focus of government policy was on improving effectiveness in the system, primarily through better administration and increased participation by poorer households, but not by means of better targeting of the poor. Neither explicit poverty targeting nor consolidation of social benefit programs were politically acceptable strategies. The Bank’s Country Assistance Strategy (2012-2016) includes better targeting in addition to improved quality of public services. The PDO includes a targeting objective and an objective of reduced burden on beneficiaries. The former objective was in conflict with government priorities for reform. The latter is interpreted as “improved access to welfare services” as explained in the ICR and by the TTL. Implicitly, this latter objective approaches government and Bank strategy for the sector.

b. Relevance of Design:

Relevance of design is rated modest.

Project design did not fully support the PDO. It included measures to improve social assistance administration by introducing single-window offices for client services and strengthening policy analysis. It did not include measures that might directly improve targeting. The implementation plan turned out to be unrealistic, with a design that overestimated the capacity of the national and sub-national welfare mechanisms to integrate new processes and to work together. The project timetable was revised, and it ultimately was transformed from a two-and-a-half-year project to an eight-year one. A detailed discussion of shortcomings in project design is presented in paragraphs 18 to 22 of the ICR. The results framework was weak, and causal links between activities, outputs and outcomes, and how these might contribute to the PDO, were absent. The sub-objective of “targeting” was not addressed. The sub-objective “burden on beneficiaries” was not defined, nor were there any indicators that directly addressed a reduced burden, however defined.


4. Achievement of Objectives (Efficacy) :

OBJECTIVE: to assist the Borrower in achieving significant improvement of the effectiveness of Ukraine’s social assistance system by better targeting of the cash benefits and the reduction of burden on beneficiaries.

The objectives statement has two sub-objectives. Achievement is assessed here for each one.

1. Better targeting of cash benefits is rated negligible.

Outputs

  • There were no outputs related to this sub-objective.

Outcomes
  • Between 2008 and 2011, the share of social assistance benefits received by poor families remained roughly unchanged at 43 percent, against a goal of increasing poverty targeting to 60 percent.

2. The reduction of burden on beneficiaries is rated modest. (Reduced burden on beneficiaries was defined as improved access to welfare services with the introduction of the single window office model.)

Outputs (outputs have, for purposes of assessment, been assigned to this sub-objective):

  • A standard single window client service office model was developed, piloted, evaluated, and formally approved.
  • 752 rayon/city welfare offices were established on the basis of the single window model.
  • Although it was planned that 27 oblast social welfare offices would become capable of carrying out analyses and monitoring of the benefit payments, this was not achieved. At the end of the project,
  • A functional review of the Department of Social Development Strategy was completed and a business plan prepared in 2006.
  • Social reports were published annually, starting in 2006 .
  • The Social Development Database was improved.
  • Two program impact evaluation studies were completed in 2008 and 2009.
  • A public information and education strategy was developed and approved in 2006.
  • 56 percent of the population was informed about the eligibility criteria for 4 benefits, and 72% were informed about eligibility criteria for 3 benefits.

Outcomes
  • Benefit application processing times were reduced from 4.5 hours in 2005 to 1 hour in 2012.
  • The number of benefits processed per month per staff of the social assistance office increased from 260 in 2005 to 540 in 2012.

The project did extend a single-window business model nationwide, which can reasonably be assumed to facilitate access to social assistance services, and thus reducing the "burden" of applying for and drawing benefits. However, there is no evidence of the impact of that measure on system participants, nor on how the measured outcomes affected beneficiaries. The absence of a functioning management information system (MIS), which had not yet been put in place by the end of the project, reduces the value of the network both for service provision and system planning. Completion of the MIS is also facing difficulties due to lack of budgets for operations and maintenance and further development, and the absence of an information technology strategy in MOLSP.

5. Efficiency:


Project efficiency is rated modest.

The PAD estimated an internal rate of return of 24.6 percent. The returns were based on estimates of savings accruing to the social assistance administration and to clients from the administrative changes introduced under the project. For the administration, gains were to be drawn from lower administrative costs, improved effectiveness of operations, better planning, and lower targeting errors. For clients, the one-window system was expected to lower transaction costs of obtaining benefits, especially for recipients of multiple benefits. There is no information provided on which to assess the correctness of the assumptions in the analysis. However, even if the assumptions were plausible, returns should have been affected by the five-year extension in project implementation, additional delays in implementing the management information system, and the absence of any improvements in targeting. All these elements would have had adverse effects on efficiency. The ICR includes no economic or financial analysis.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
24.6%
100%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


There were major shortcomings in project relevance, achievement of objectives, and efficiency. Objectives were only partly relevant to Government and Bank strategies for the sector and rated modest. With weak causal links between activities, outputs and outcomes, and between them and the PDO, and unrealistic implementation designs, relevance of design is rated modest. Efficacy of the first sub-objective, targeting, is rated negligible in the absence of relevant project activities. Efficacy of the second sub-objective, burden, is rated modest based on outputs and outcomes that only had a tenuous relationship to the sub-objective. Efficiency is rated modest, supported by the potential positive effects of the one-window system, but downgraded in the absence of any post-project economic or financial analysis.

a. Outcome Rating: Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

At the time of project closing, an operational single-window service model was in place. However, some aspects of the model had not been completed. Notably, the innovative information system was not in place: While it had been tested in one region, it was still suffering from shortcomings: documents regulating the use of the system were still to be developed; and staff had not been trained in the use of the system. Moreover, the Government had not allocated necessary financial resources to complete the process, or for operations and maintenance. A fully operational MIS would enhance efficiency and effectiveness in operating, monitoring, evaluating, and adjusting the functioning of the social assistance system, and it would facilitate poverty targeting, programs consolidation, and policy development in the future.

a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:

Addressing issues in the social assistance system was strategically relevant. Social assistance took a significant share (some 4 percent) of GDP, it was fragmented and uncoordinated, and it paid little attention to poverty targeting. In developing the project, the team was able to draw on extensive prior work in the sector and on a tested single-window design that had been developed by the British development agency, DfID, and that was subsequently piloted by the Bank team under a Project Preparation Facility. Against this background, and a relatively long gestation period (preparation took four years), as well as close collaboration with the Ukrainian authorities, the project still came to face a number of issues during implementation that might have been avoided, or at least mitigated, with a more thorough preparation. The possibilities for introducing better poverty targeting into the system were overestimated. The Borrower’s implementation capacity was exaggerated: the project was negotiated for a two-and-a-half-year implementation period, but had to be extended three times, ending up as an eight-year project. This would have repercussions on the procurement process, especially for information technology equipment, and ultimately postpone implementation of the automated MIS. It would also affect disbursements that ran behind schedule for much of the early part of the project. Additional issues of implementation that might have been avoided or mitigated during preparation related to budget processes that created uncertainties in the project's flow of funds; insufficient preparation for introducing a new and innovative financial management system; and different local start-up conditions for setting up single-window offices. Page 16 of the ICR describes these issues in greater detail. Finally, while quantitative economic analysis was undertaken, it did not include any discussion of fiscal/financial implications of the project.

Quality-at-Entry Rating: Unsatisfactory

b. Quality of supervision:

While supervision benefited from proximity – key members of the team were located in Ukraine – the project nevertheless experienced implementation difficulties that led to consecutive extensions of the closing date. Still, interaction was intense, taking the form of implementation support that focused on addressing problems that at least to some extent should have been captured during the four years it took to prepare the project. Under the circumstances, the supervision team was reasonably successful in addressing the technical, knowledge, and financial problems that arose during implementation. However, it did not use any of the three extensions of the project to address weaknesses in design that would ultimately affect the project outcome: inconsistencies between design and the PDO, a weak results framework and a modest M&E design. The team is in the process of building on the project to prepare a follow-on project under an ECA Development Trust Fund. That project will include strengthening the functionality of the single-window offices, drawing on the incorporation of the management information system.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The government was committed to pursuing increased effectiveness of the social assistance system through better administration of existing benefits. Commitment to consolidating a fragmented system or better targeting of benefits at poor households and individuals appears to have been less strong, which is likely to have reflected the absence of broad political and popular support for such changes to the system. Ultimately, this would lead to failure to achieve project objectives. While the project was prepared in close collaboration between the government, the implementing agencies, and the Bank, none of the participants appear to have had a realistic grasp of implementation capacity on the Borrower’s side as reflected in the initially short (two and a half year) implementation period. Project implementation also suffered disruptions during implementation, as the Minister of Social Policy changed eight times, and together with the Minister, officials directly involved with the project also changed. On the positive side, during implementation, the government did adopt a traditionally complicated budget system that would, despite initial difficulties, ensure annual budget allocations and timely financial flows throughout the system.

Government Performance Rating: Unsatisfactory

b. Implementing Agency Performance:

The implementing agency, the MOLSP, was committed to achieving the PDO, including better targeting of benefits to poor households. The latter was reflected in efforts to improve the targeting system, even though these efforts had to be dropped as they were seen as politically too sensitive. Despite numerous changes in leadership that may have contributed to slower-than-expected implementation, the MOLSP was able to collaborate with regional and local authorities to put into place single-window offices and provide the necessary counterpart financing. Project monitoring allowed for adequate measuring of project processes and outputs, as well as improvements in institutional effectiveness. Implementation issues (relating to the establishment of offices, procurement, and financing) were resolved in what proved to be effective, albeit slow, collaboration between the ministry, sub-national authorities, and the Bank. No fiduciary issues arose during implementation, with the exception of slow disbursements, which reflected problems in design and absorptive capacity.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The project developed capacity in the State Statistics Committee to collect survey information on social assistance that would enable measurement of targeting efficiency and poverty reduction impact of each program. Monitoring design for the project focused on improved targeting and more efficient program application and benefit processing. While the targeting was directly relevant as an outcome measure, it was not directly supported by any project activities; and institutional M&E spoke to an internally more efficient administration, but hardly to beneficiary well-being. The impact of the single window, which actually might have formed the basis for some results-based analysis, was not measured. That said, data collection methods were appropriate. They drew on administrative data gathered in welfare offices, a perennial national household budget survey, and complementary client surveys.

b. M&E Implementation:

All indicators specified in the PAD were measured and monitored throughout project implementation. The database was continuously improved throughout the project period, albeit without the benefit of an automated management information system, which was not completed by project closing. Again, had the right things been measured, M&E would have been significantly more effective.

a. M&E Utilization:

Administrative information flows between welfare offices and the central MOLSP allowed comparing performance of welfare offices and adjustments in their operations during project implementation. There is no evidence of strategic changes in social assistance programs. That said, the project laid the foundation for a system of monitoring, evaluating, and adjusting policy based on information. This observation is reinforced by the institutionalization of M&E of social assistance programs through a government decree issued at the end of 2012.

M&E is rated modest, rather than negligible, because the project did help develop a monitoring and evaluation system that has been adopted by the government for regular monitoring and evaluation of social assistance programs.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

The environmental impact of the Project was rated C. The civil works supported by the operation were not expected to have significant negative effects on Ukraine’s environment or natural resources (ICR, page 19).

b. Fiduciary Compliance:

Financial management. Disbursements lagged behind projections, especially during the first years of project implementation. This reflected delays in (i) hardware and software installation (almost 95 percent of project costs) due to complications in the construction and rehabilitation of welfare offices; and (ii) applying unfamiliar financial management procedures. The latter channeled Bank funds through the State Treasury Service, allowing for simultaneous disbursement at the central, regional, and local levels. While a useful innovation, it turned out to require a steep learning curve at sub-national levels. Project audits were delivered on time and they were unqualified.

Procurement. Procurement became slower than expected. Initially, national procurement rules were to be followed for civil works. However, with a change in national procurement rules, Bank rules were introduced for all procurement under the project. This proved difficult for regional and local staff, and even with support from a strengthened central project implementation unit, delays were experienced. In addition, some fraud and corruption in procurement was substantiated, but did not lead to cancellation of funds.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Unsatisfactory
Unsatisfactory
There were major shortcomings in project relevance, achievement of objectives, and efficiency. Objectives were only partly relevant to Government and Bank strategies for the sector and rated modest. With weak causal links between activities, outputs and outcomes, and between them and the PDO, and unrealistic implementation designs, relevance of design is rated modest. Efficacy of the first sub-objective, targeting, is rated negligible in the absence of relevant project activities. Efficacy of the second sub-objective, burden, is rated modest based on outputs and outcomes that only had a tenuous relationship to the sub-objective. Efficiency is rated modest, supported by the potential positive effects of the one-window system, but downgraded in the absence of any post-project economic or financial analysis.  
Risk to Development Outcome:
High
High
 
Bank Performance:
Moderately Satisfactory
Moderately Unsatisfactory
IEG assesses that there was misjudgment in project design by the preparation team, and while supervision successfully addressed routine issues, weaknesses in project design that affected outcomes were not addressed. Ultimately, this led to failure to achieve project objectives. 
Borrower Performance:
Moderately Satisfactory
Moderately Unsatisfactory
IEG emphasizes the effects of the unstable political environment and the unrealistic ambitions reflected in the project schedule, of which the Government should have better grasped. Moreover, the Government's reluctance to pursue poverty targeting was a contributing factor to the unsatisfactory project outcome. 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The following lessons are drawn from the ICR (pp. 27-29):

Projects involving large IT components should include a sector-specific IT specialist to facilitate the incorporation of the hardware and software with overarching sector-related issues.

Sustainability of software, hardware, and knowledge to use equipment must be addressed throughout the project cycle. This includes development of budgets for sustainable operations, data migration, testing systems in a full production environment, and securing human resources for system maintenance.

Implementation of any kind of uniform system across a country with great variance in local capacity and attitudes requires strong oversight and control mechanisms at the central level.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:

While the evidence presented in the ICR is of good quality, it does not sufficiently draw the consequences of that analysis regarding the performance of this project, in particular, the results achieved. The ICR is internally consistent, follows the guidelines, and has been kept at reasonable length.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions