Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Social Policy & Community Social Services


  
1. Project Data:   
ICR Review Date Posted:
03/06/2006   
PROJ ID:
P008539
Appraisal
Actual
Project Name:
Social Policy & Community Social Services
Project Costs(US $M)
 12.31  15.19
Country:
Lithuania
Loan/Credit (US $M)
 3.7  3.61
Sector, Major Sect.:
Central government administration, Other social services,
Law and justice and public administration; Health and other social services
Cofinancing (US $M)
 4.3  2.84
L/C Number:
L4135      
   
Board Approval (FY)
  97
Partners involved
Sida, Netherlands 
Closing Date
01/10/2002 05/31/2005
         
Evaluator: Panel Reviewer: Division Manager: Division:  
Howard Nial White
Nalini B. Kumar Alain A. Barbu IEGSG

2. Project Objectives and Components:

a. Objectives

The Social Policy and Community Social Services Development project supported the Government of Lithuania in providing social security to the population at an affordable cost through institutional development activities to improve: (i) the capacity to analyze social policy at all stages; and (ii) community social care services.

The proposed project would assist the Government in the following areas: (a) developing the capacity of the Ministry of Social Security and Labor (MSSL) to design, implement, monitor and evaluate social safety net policies; (b) reducing institutionalization of groups at risk (the handicapped, the elderly, etc.) through the development, testing and replication of new community-based social service models; and (c) developing MSSL project management capacity.

b. Components (or Key Conditions in the case of Adjustment Loans):

1. Social Policy Development (Planned: US$2.60 million; 24% of base cost; Actual: US$ 1.75 million, 12%). This component sought to improve the allocation of human and financial resources within the MSSL through a strategic planning approach, so that scarce organizational resources would be used more effectively and priority policy development and evaluation addressed. The component also aimed to strengthen the capacity of the Ministry to develop workable social policy solutions to be presented to the MSSL's clients. It also supported the local capacity to prepare high quality social policy analyses.

2. Community Social Service Pilot Projects (Planned: US$7.08 million, 66% of base cost; Actual: US$ 12.4 million, 82%). This component included three subcomponents: (a) establishment of multi-service centers for the elderly, families with children and people with disabilities, including support to three pilot services in Vilnius, Siauliai and Svencionys serving various vulnerable groups; (b) establishment of education centers for children with disabilities, including support to three pilot centers in Anyksciai, Moletai and Utena aiming to prevent handicapped children from being placed in institutions by developing services in the community; and (c) monitoring and evaluation.

3. Project Management (Planned: US$1.02 million, 10% of base cost; Actual: US$ 1.04 million, 7%). This component supported the implementation of the project activities through the development of the Project Coordination Unit (PCU) thus enhancing the project management capacity in the MSSL. The PCU was established within the MSSL and served as a Secretariat of the Project Steering Committee, chaired by the Minister. The main responsibilities of the PCU were: (a) to coordinate, monitor and evaluate project activities; (b) to manage and keep records of all procurement operations; (c) to prepare and distribute periodic reports to the relevant government authorities and to the Bank in an agreed format.

c. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Projects costs at appraisal were US$ 12.31 million, and actual costs US$ 15.19 million, a cost overrun of 23%. The excess costs were entirely on the second component (pilot projects) and met by increased government contributions.

The SAR mentions that US$4.29 million co-financing had been obtained; US$3.44 million from Sweden for component 2 and US$0.85 million from the Netherlands for component 1. The ICR reports a lower figure for planned cofinancing of component 2, but includes US$0.25 million for component 3. There was no actual cofinancing for the third component and that for the other two components fell short by US$ 0.5 million for each component (or US$ 1 million for component 2 taking the original SAR figure); though the ICR does not say why this was.


3. Relevance of Objectives & Design:

The project objectives were consistent with the Government's new policy initiatives and they were in line with the Lithuania Country Assistance Strategy at the time and remain relevant.

The design of project components was clearly linked to the project objectives. The project focused on the Government’s priorities, i.e. re-orienting the social safety net through institutional and capacity development that allowed taking well informed, technically and fiscally feasible social policy decisions, and testing and introducing new social assistance forms and approaches in addressing social service needs of vulnerable populations. Adequate attention was given to project management and coordination arrangements at an early stage of the project.


4. Achievement of Objectives (Efficacy) :

Improve the capacity to analyze social policy at all stages: Substantial achievement

The MSSL has developed into an institution that has the capacity to design, evaluate and monitor social policies, is proactive and transparent in areas of responsibility and efficient in communicating that to all counterparts at central and local government levels and the public at large. Several policy instruments that have been developed at the initial project stages - Annual Strategic plans, Annual Social Reports, Policy Evaluation Fund - continue to be an important part of the MSSL activities. The project activities have significantly increased the human capacity at the MSSL.

Improve community social care services: Substantial achievement

The early establishment of three social service pilot centers in 1998 (the PPF was used to start these pilot centers) and the quality of services provided there have been considered as best practice in the region, and as such worked as a key factor in establishing demand for similar centers throughout the country and the Government’s response in developing and implementing a national Social Service Infrastructure Program. Technical assistance provided under the project laid a solid ground for further institutional and social policy development in the social service area.


5. Efficiency:

There are two aspects to efficiency. First, the project supported the move to a more efficient basis for delivering services, with unit costs in the Bank-supported day care facilities being lower than those in existing residential facilities. The second aspect is the efficiency with which activities were carried out. Most activities were completed according to schedule. The two negative points were: (1) the project was extended three times for a total of 3.5 years, mainly to allow for the implementation of the Strategic Partnership Program, whilst other activities were completed closer to schedule. Therefore extra supervision costs were incurred whilst this activity was completed; and (2) the apparent cost overruns of 23% overall (75% for the second component). The region subsequently provided satisfactory explanations for these and, on that basis, efficiency is rated substantial.

6. M&E Design, Implementation, & Utilization:

M&E were central to the project, as monitoring and evaluation of the evolution of social policy, including reforms supported by the project were at the heart of the first objective and built into the first component. These activities were carried out in a satisfactory manner.

In addition, all project components were to be monitored by the Steering Committee and the PCU. The SAR included a log frame (Annex A, page 35) with a well defined set of indicators for the first two components, including measurable indicators with quantified targets. These indicators were developed by the project working groups, and Government confirmed agreement on these indicators during negotiations.

The M&E of reforms carried out was successful, and has informed government policy. The degree of implementation of monitoring of project components is not clear from the ICR since some, but not all (e.g. client satisfaction) of the SAR indicators are reported on in the ICR. Importantly, the borrower ICR provides unit cost comparisons for existing residential facilities and Bank day care facilities, demonstrating the considerable cost savings from the latter.

The ICR is, however, critical of the monitoring, including a lesson learned that “although the project was clear about expected outcomes, the key performance indicators were not very useful to monitor project performance. To monitor the project progress there should be a limited number of monitoring indicators that, preferably, are part of the regular business of the implementing agency and do not require special arrangements such as surveys, analysis etc.” This review does not share these reservations. The performance indicators were well defined across the log frame; the criticism that the targets were met seems to imply satisfactory performance rather than shortcomings in the indicators. Since there was a separate PCU it is not surprising that some indicators fell outside of government’s usual data collection. Whilst it is desirable to use existing survey data for evaluation purposes where feasible, many interventions, especially those with narrowly defined target groups as was the case here, require special surveys.

7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative):

None.


8. Ratings:
ICR
ICR Review
Reason for Disagreement/Comments
Outcome: 
SatisfactorySatisfactory
Institutional Dev.: 
SubstantialSubstantial
Sustainability: 
Highly LikelyHighly Likely
Bank Perf.: 
SatisfactorySatisfactory
Borrower Perf.: 
SatisfactorySatisfactory
Quality of ICR: 
Satisfactory

NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness.

9. Lessons:

The ICR identifies a number of lessons:

  • Comprehensive and participatory project preparation that includes substantial analytical work, initial development of policy framework and basic principles of institutional and organizational framework helps ensure high quality implementation. The project benefited from good preparation that created real ownership and determined early success of the project.
  • It would be advantageous if completed components could be formally closed, rather than the whole project having to be extended on account of one lagging component. In the case of this project implementation of the renewed Strategic Partnership Program (SPP) lagged behind other activities so that the project was extended three times and significantly increased the project supervision costs.
  • For project activities where cooperation between different levels of government is essential, efforts need to be spent initially to build real partnership that is based on mutual understanding about the objectives and benefits of the joint program.
  • Capacity building for policy analysis, monitoring and evaluation takes time. Only programs that cover several years can achieve sustainable results and they should provide a mix of analytical and practical lessons.

10. Assessment Recommended?  Yes

          Why?  Apparently successful project. One of a number of social protection projects in ECA, forming a potential cluster.

11. Comments on Quality of ICR:


The ICR is clearly written and details the progress and (limited) problems of the project. It draws a number of general lessons. There are, however, two shortcomings: (1) the cost overrun (23%) is quite substantial, but the ICR makes no attempt to explain it (satisfactory explanations were subsequently provided by the region), and (2) more explicit attention could have been paid to M&E of the project itself.

(ES-Rev4B-Dec/05)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions