Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Social Investment Fund 2 Project


  
1. Project Data:   
ICR Review Date Posted:
03/11/2014   
Country:
Moldova
PROJ ID:
P079314
Appraisal
Actual
Project Name:
Social Investment Fund 2 Project
Project Costs(US $M)
 29.2  65.5
L/C Number:
C3931, C4555
Loan/Credit (US $M)
 20.0  45.3
Sector Board:
Social Protection
Cofinancing (US $M)
 7.6  18.0
Cofinanciers:
SIDA, Soros, Communities
Board Approval Date
  06/17/2004
 
 
Closing Date
03/31/2010 03/31/2013
Sector(s):
Other social services (40%), Primary education (20%), Roads and highways (15%), Energy efficiency in power sector (15%), Water supply (10%)
Theme(s):
Other social protection and risk management (33% - P) Rural services and infrastructure (17% - S) Rural policies and institutions (17% - S) Participation and civic engagement (17% - S) Other accountability/anti-corruption (16% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Hjalte S. A. Sederlof
Judyth L. Twigg Lourdes N. Pagaran IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The Project Development Objective (PDO) as set out in Schedule 2 of the Financing Agreement was to contribute to the implementation of the Borrower’s EGPRS (Economic Growth and Poverty Reduction Strategy) by empowering poor communities and vulnerable population groups to manage their priority development needs.

    The same PDO was included in the Project Appraisal Document (PAD).

    The PDO was revised in January, 2010. At that time, additional financing (Second Additional Financing) was introduced to help mitigate the effects of economic recession driven by the international economic downturn, which was having depression-scale effects on rural incomes. The original PDO was maintained, and an additional objective was added: to contribute to employment and wage incomes in selected poor rural communities during the current economic contraction and during the recovery.

    Key indicators were introduced to support the new objective. Key indicators for the original objective already had been modified in 2006, and were modified again in 2009. The new PDO and a full set of revised indicators were approved by the Board on January 26, 2010.

    The project is assessed here against both objectives.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 01/26/2010

c. Components:

Original components

Original component 1: Community development (total estimated cost at appraisal US$ 22.5 million; actual cost US$ 25.1 million).
This component was to provide or rehabilitate basic services in poor rural communities and small towns. It was to do so by co-financing sub-projects with local governments and communities. In addition to basic services, it would support capacity building to strengthen local and community planning, operations and maintenance.

Original component 2: Social care services development (total estimated cost at appraisal US$ 3.3 million; actual cost US$ 3.5 million).
This component was to include: (i) preparation of a master plan for social care services development; (ii) social care services sub-projects; and (iii) capacity building for public authorities, service providers and social workers. Beneficiaries were to be biological and social orphans, people with disabilities, the elderly, families and individuals in crisis, etc. Component activities were to be undertaken in six rayons (districts) based on their expressions of interest and defined selection criteria. These initial districts were to serve as pilots for subsequent expansion.

Original component 3: Capacity development, communication, monitoring and evaluation (total estimated cost at appraisal US$ 1.9 million; actual cost US$ 0.8 million).
This component was to include (i) capacity building to support transfer of learning and experiences from local communities to government institutions; (ii) development of monitoring and evaluation mechanisms for monitoring implementation of the EGPRS; and (iii) communication, dissemination, and replication of best practices .

Original component 4: Project management (total estimated cost US$ 1.5 million; actual cost US$ 1.0 million).
This component was to include support for the operations of the implementing agency, the Moldova Social Investment Fund (MSIF).

Revised components
Additional financing was introduced twice, once on May 4, 2009 without changing the PDO, and another time, on January 26, 2010, when the PDO was revised.

The first additional financing (total estimated cost US$ 5.9 million; actual cost US$ 6.9 million) was introduced to mitigate the effects of a severe drought on poor rural households. It intensified activities under Original Component 1 and strengthened project management. The difference in estimated and actual costs reflects an increase in the community contribution to sub-projects.

The second additional financing (total estimated cost US$ 25.2 million; actual cost 28.4 million) was introduced to mitigate the effects of the 2008-2010 international economic crisis. It also intensified activities under Original Component 1, but also aimed at increasing their labor intensity. Again, the difference between the estimated costs and actual costs reflects additional contributions by local communities.

Other significant changes

During the introduction of the two Additional Financings, increased emphasis on implementing sub-projects was shifted from community-based committees to village and small-town governments, reflecting the enhanced role these government levels were coming to play in linking the MSIF to community operations, and the opportunity this created to developing sustainable institutions to promote community initiatives. Accordingly, increased focus was placed on relevant training at the level of local government in procurement and financial management.

Also, in 2012, school rehabilitation under the project was limited to an agreed list of schools, to eliminate the risk of rehabilitating schools that potentially were to be closed under a national school consolidation program.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project cost and financing, including the Borrower contribution, is set out in the following table:

Project costs
Appraisal estimate (US$ mln) Actual (US$ mln)
Original Add’l 1 Add’l 2 Total Original Add’l 1 Add’l 2 Total
Totals 29.2 5.9 25.2 60.2 30.4 6.9 28.4 65.7
Sources of funds:
IDA 20.0 5.0 20.0 45.0 20.7 5.3 19.3 45.3
SIDA (1) 3.7 0.0 2.7 6.4 4.2 0.0 2.4 6.6
Soros Foundation (2) 0.2 0.0 0.0 0.2 0.0 0.0 0.0 0.0
Local communities 3.7 0.7 2.5 6.9 3.8 1.4 6.0 11.2
Borrower (3) 1.5 0.2 0.0 1.7 1.5 0.2 0.7 2.4
Total 29.2 5.9 25.2 60.3 30.2 6.9 28.4 65.5

(1) SIDA= Swedish Agency for International Development. US$ 3.7 million financed social care services,
and 2.7 million was joint financing with the Bank.
(2) The Soros Foundation initially committed to participate, but then chose to partner with another organization. This was never corrected in MSIF accounts, and therefore there is a discrepancy between the ICR's numbers and those in this ICRR.
(3) The Borrower contribution was in the form of joint financing

Dates. The project was approved on June 17, 2004. Additional financing was provided twice, on March 24, 2009 and January 26, 2010. The first additional financing was introduced to counteract the adverse effects of a drought on poor rural households, while the second was to mitigate the effects of the international economic crisis on incomes. The closing date was originally set for March 31, 2010. It was extended once to accommodate the second additional financing, to March 31, 2013.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Relevance of objectives is rated high for both the original and additional PDOs.

The PDO focused on promoting community development and social care, two interventions that directly supported the EGPRS, introduced in 2004, and a subsequent National Development Strategy (NDS) stretching over the period 2008 to 2011, and subsequently extended to 2020. There, community development was seen as an important tool for flexibly addressing chronic poverty as well as vulnerability to occasional systemic shocks. Social care services were seen as a means of dealing with social exclusion and family crisis. The PDO is also aligned with the Bank’s current Country Partnership Strategy (CPS) that seeks to minimize social risks, including by strengthening social protection systems. The Additional Financing that was introduced during implementation was also in line with Government and Bank strategies, responding to the effects of first climatic and then economic shocks to the Moldovan economy.

b. Relevance of Design:

Relevance of design is rated substantial for the original PDO and modest for the added PDO.

Project design was directly relevant to the original PDO. The underlying project logic, as articulated in the results framework, was sound. It included community development activities (socio-economic sub-projects and related capacity building) in poor rural communities and small towns; social care services focusing on the family and individuals at risk; and capacity building to strengthen local governments in backstopping community and family initiatives. The community development design drew on the experience gathered under a first Social Investment Fund Project (P044840) that closed as the project under review started, in particular the emphasis placed on including the community in decision making on all aspects of the sub-project cycle.

Project design was also directly relevant to the new objective that was introduced with the second Additional Financing, as it allowed the project to be adapted to changing circumstances. However, in practice this turned out to be less so: while the emphasis of sub-project design was placed on greater labor intensity, the technical designs for sub-projects turned out to be more difficult to adapt to more labor-intensive approaches than expected (see Section 4).


4. Achievement of Objectives (Efficacy) :

Objective: The original objective was to contribute to the implementation of the Borrower’s EGPRS (Economic Growth and Poverty Reduction Strategy) by empowering poor communities and vulnerable population groups to manage their priority development needs. A second objective was added with the Second Additional Financing in 2010: to contribute to employment and wage incomes in selected poor rural communities during the current economic contraction and during the recovery.

Efficacy is rated against both objectives.

Objective 1: to contribute to the implementation of the Borrower’s EGPRS (Economic Growth and Poverty Reduction Strategy) by empowering poor communities and vulnerable population groups to manage their priority development needs.

Efficacy for Objective 1 is rated substantial.

Outputs

    • A total of 854 community infrastructure sub-projects was completed (443 under the original credit; 74 under the First Additional Credit; 307 under the Second Additional Credit; and 30 under SIDA financing).
    • Social care services were introduced on a pilot basis in six districts under SIDA funding.
Outcomes

Note that the targets are estimates of results that could be expected, based on the sub-project approval documents. Actual outcomes that exceed the estimated targets reflect higher than expected effectiveness in the use of sub-project outputs.

All baselines are for 2004; outcomes and targets are for 2013.

Education and health

    • School attendance in participating school sub-projects rose by 3 percent, from a baseline of 92 percent to 95 percent in 2013, in line with the target.
    • Learning in rehabilitated schools (measured as percentage of students passing year-end exams) rose by 4 percent, from a baseline of 93 percent to 97 percent in 2013, compared to a target of 96 percent.
    • The share of participating schools with new educational programs increased from a baseline of 9 percent to 65 percent in 2013, compared to a target of 90 percent.
    • The number of cases of respiratory and infectious diseases in communities with school water and gas sub-projects fell from a baseline of 180 cases per 1,000 per year to 98 in 2013, as compared to a target of 120.
Community behavior
    • Consumption of coal and fuel wood for heating among households participating in gas connection sub-projects in project communities decreased from a baseline of 85 percent to 45 percent, against a target of 24 percent.
    • Participation by community members in community development in project communities increased from a baseline of 28 percent to 70 percent (no specific target was set).
    • Project communities that had benefited from MSIF support undertook 244 sub-projects on their own outside the MSIF framework, against a target of 107.
    • 53 percent of survey respondents indicated that their capacity to manage priority community development needs had improved because of MSIF training and the experience of sub-project implementation, against a target of 60 percent. An earlier survey, in 2010, had indicated that this was the case with 67 percent of respondents.
    • 35 percent of surveyed respondents expressed increased satisfaction with local governments’ way of managing community priority needs, against a target of 30 percent. An earlier survey, in 2010, indicated a 49 percent rate of satisfaction.
    • 507 sub-projects received higher community contributions than required. (However, this number includes non-project sub-projects as well as project ones.)
    • Beneficiary associations continued to operate two years after sub-project completion in 357 communities, as compared to a target of 90 communities.
    • 12 government and non-government institutions adopted best practices from the MSIF.
Social care services
    • 31,621 persons benefited from community-based social care services (no targets were set).
    • 80 percent of surveyed civil society representatives in coordinating committees indicated that their involvement in planning and provision of social care services had improved, as compared to a target of 60 percent.

Other
    • The project had a demonstration effect: (i) 384 sub-projects were financed through other donor grants channeled through the MSIF: the European Union (EU), Kreditanstalt fur Wiedraufbau (KfW), and the Japanese Social Development Fund (JSDF); (ii) following the social care pilot, the EU financed 15 social care services, and the Government began construction of a national network of social care services.
To the extent that increased participation in community activities defines empowerment, survey findings indicate that the objective was achieved, as the majority of members of poor communities state this to be the case. At the same time, the capacity for community development (making practical use of empowerment) was enhanced, as reflected in new initiatives taken outside the project, new processes adopted drawing on MSIF practices, and better communication with local and national government in pursuing community priorities. The introduction and expansion of social care services should have begun to directly address the needs of vulnerable families, although the effectiveness of these services remains unclear.

In some instances, the impact of the project at the community level appears to have declined over time (community-level management capacity, satisfaction with local government management). The project team attributes this to changes in government following local elections prior to the 2010 survey.

Objective 2: to contribute to employment and wage incomes in selected poor rural communities during the current economic contraction and during the recovery.

Efficacy for Objective 2 is rated modest.

Outputs

    • 307 labor intensive sub-projects were supported.
    • An estimated 149,000 person-days (21 percent female) were expended on labor-intensive sub-projects.

Outcomes

Baselines are measured for 2009 and outcomes/targets for 2013.

    • Labor intensity increased in the main sub-project groups, measured by the share of wages in the sub-project contract budget: building construction from a baseline of 23 percent to 27 percent, not meeting the target of 40 percent; water and sanitation from a baseline of 15 percent to 22 percent, not meeting the target of 30 percent; village roads from a baseline of 12 percent to 25 percent, meeting the target of 24 percent; and gas lines from a baseline of 10 percent to 28 percent, exceeding the target of 20 percent.
    • The increase in wage incomes in communities that received sub-project support under the Second Additional Financing was an estimated 24 percent over the period 2010-2011, not meeting the target of 40 percent.
    • The increase in consumption expenditures for the same cohort over the same period was an estimated 18 percent, not meeting the target of 22 percent.
While labor intensity, as measured by the share of wages in the sub-project budget, appears low, wages did increase, even double in some instances, raising wage incomes by some 25 percent, and positively affecting household consumption as well as household savings. Still, increases in wage incomes and consumption spending did not achieve the targets. A greater emphasis on labor intensity, raising the share of labor in total costs to a more conventional 40-50 percent, might have significantly increased incomes and household consumption beyond the current numbers. Importantly, no information is provided on employment outcomes, which was a key part of the objective.

5. Efficiency:


Efficiency under the project is rated substantial.
The PAD included rate of return analyses for three selected sub-project types, where sufficient information was available to do so and reasonable assumptions could be made. For schools, it varied between 12 and over 60 percent; for gas supply it was estimated at 30 percent; and for water supply systems, at 100 percent. The ICR assesses efficiency based on comparing management and construction component costs with selected similar projects in the ECA Region. It notes that management costs at 4.8 percent were half the average for nine similar projects in ECA. In comparing project-supported infrastructure sub-projects with other donor- and Government-financed projects, the results suggest that construction costs for the MSIF project systematically were lower than the comparators. Efficiency under the second Additional Financing was modest, reflecting the relatively low levels of labor intensity that were generated by the additional investment, as compared to levels generally sought under labor intensive works projects. Missed targets for wage income and consumption also point to modest efficiency. There is no analysis of the fiscal-financial implications of project initiatives, which might pose risks to the sustainability and future development of the MSIF approach and the social care network.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Rating against the original PDO:
The relevance of the PDO is rated high: it addressed a central feature of the government’s ongoing development plan, and one that accords with the CPS. The relevance of design is rated substantial: the components directly supported the PDO and the aims of Government and the CPS. Efficacy is rated substantial, reflecting positive intermediate and final outcomes in all areas of intervention. Efficiency is rated substantial, based on rates of return and cost effectiveness analysis in the ICR.

The outcome under the original objectives is rated satisfactory.

Rating against the original plus additional PDO:
The relevance of the PDO for the second Additional Financing Credit is also high, as it supported the mitigation of short-term shocks that also were in line with Government and Bank strategies. The relevance of design to the PDO is rated modest, reflecting shortcomings in the technical design of sub-projects. Achievement of the original project objective is rated substantial, but achievement of the additional project objective is rated modest as a result of the weak results in terms of labor intensity, incomes, and consumption, and the lack of data on employment.

Because of the importance of the additional objective to the project under the Additional Financing, the outcome under the revised objectives is rated moderately unsatisfactory.

43.3% of disbursements were made under the revised objectives. Following IEG/OPCS guidelines, the overall outcome rating is determined by weighting the outcomes under each set of objectives according to the percentage of disbursements under those objectives.

Original objectives: 0.567 * 5 (Satisfactory) = 2.84

Original plus revised objectives: 0.433 * 3 (Moderately Unsatisfactory) = 1.30

Total: 2.84 + 1.30 = 4.14, which rounds to 4, for an overall outcome rating of Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The institutions that were supported under the project continue practices that already were introduced under the first MSIF project. New initiatives, both in terms of empowerment and social care services, appear to have structures that are likely to be maintained: citizen participation is being promoted by the government, and a national strategy for expanding and strengthening social care services has been put in place. Some uncertainty is being introduced by the Government not allocating a budget for maintaining MSIF, on the assumption that donors would continue to finance the organization. In fact, considerable donor support appears to be available for expanding MSIF-type activities and social care.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:

The project supported initiatives that were relevant to the Government and the Bank. In designing the project, it drew on the experience of the earlier first MSIF project. The project team, which had supervised that project, also was able to draw on its prior experience in working with Government and implementing agency officials to ensure adequate implementation capacity on the ground. This would later be reflected in a relatively problem-free implementation. Risk was assessed as substantial, in particular reflecting concerns about consistent government commitment to MSIF investments and rent-seeking by local elites. Project design included measures to reduce these risks by ensuring broad government and community participation. Subsequent implementation reports indicate that these measures were successful. While a monitoring and evaluation strategy was designed that would allow necessary information and feedback processes, the initial results framework would turn out not to be sufficient to measure progress or achievement of project outcomes.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

Apart from the indicator-related difficulties, which were gradually corrected, no significant problems arose during project implementation. The Bank team undertook semi-annual supervision missions that included field visits and focused on both empowerment features and sub-project quality. These also included procurement post-reviews and safeguard reviews. Significantly, the team was able to anticipate potential school closings and guide the project towards more permanent facilities; it was also able to accelerate sub-project implementation during the Second Additional Financing period, at the same time informally influencing the increased labor emphasis in those sub-projects.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

While there initially seems to have been concern about Government commitment to the project, reflected in complaints about disorganized meetings, the Government promptly addressed these concerns and proved in fact to be strongly committed to the project. This commitment was sustained through a change in the administration following elections in 2010. At the local level, commitment remained strong throughout, reflected in substantial community contributions to sub-projects and active application of participatory techniques throughout the project cycle.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:

The MSIF was able to take on a workload that may not have been anticipated at project start-up, as donors contributed financing for a number of additional sub-projects, channeled through MSIF, which provided the overall management and administration of sub-project implementation according to Bank guidelines. In part, this reflected experience from the first project, where MSIF capacity initially had been stretched. This time, that did not happen.

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

M&E design emphasized measurement of outcome and development impact, project outputs, and feedback into the project. Outcome and impact monitoring was to include impact and beneficiary assessments. Project performance and output monitoring was to include procurement and technical reviews, financial audits, and semi-annual progress reports based on output indicators. Community level monitoring was to take place through participatory M&E with a list of indicators to be monitored.

The underlying project logic, as articulated in the results framework, was straightforward, but the accompanying indicators were not. They presented a mixture of output and outcome indicators, and some that were to prove difficult to interpret and/or measure. The situation improved with subsequent revisions to indicators, but overall, lapses in baselines, milestones and outcomes made project monitoring unnecessarily difficult.

b. M&E Implementation:

The M&E system drew on the MSIF management information system (MIS), semi-annual progress reports, special reports, and norms and standards for construction. The results framework was modified during implementation to better capture achievement of development objectives, in particular introducing a stronger emphasis on indicators for empowerment. The MIS predominantly collected and analyzed output data on performance. Outcome data were collected and analyzed by consultants through special data gathering exercises.

a. M&E Utilization:

The M&E system was designed to allow assessment of project outcomes. This was used during implementation to change the emphasis in community development towards greater engagement of village and local governments. With the revisions in the logical framework, outcome indicators were increasingly introduced.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

The environmental category for the original project and the two Additional Financings was "Financial Intermediary," since the project financed investments selected by communities and not determined in advance. The Project Operation Manual included environmental guidelines and a checklist for evaluating environmental and other safeguards issues for each proposed community infrastructure sub-project. A safeguards specialist regularly reviewed the project, including visiting a sample of infrastructure sub-projects. These reviews did not find any significant issues with compliance.

b. Fiduciary Compliance:

Financial management. The fiduciary functions of the project were performed by the MSIF, which had already successfully implemented the first SIF project. All reports were submitted on a timely basis and found satisfactory. The financial audit reports were all received on time, with the exception of the last one, which was received with a delay of four months because of complications in procuring the auditor. The opinions in the Management Letters were “unqualified” in regard to the project’s financial statements.

Procurement. Procurement of all works, goods and technical services under the project followed the procurement guidelines.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
Outcomes for the project under the original and original+revised PDOs wereSatisfactory and Moderately Unsatisfactory, respectively, resulting in a weighted average of Moderately Satisfactory. 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Satisfactory
Moderately Satisfactory
The initial results framework was not adequate to monitor progress and assess achieved outcomes. 
Borrower Performance:
Satisfactory
Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
From the ICR (pp 14-15):
    • Implementation of the MSIF II project confirmed that the public is willing to participate in local development, including volunteering time and money for sub-projects. It also confirmed the value of community contributions in gaining commitment to maintaining sub-projects, and in teaching communities to raise funds for their own development projects.
    • Clear criteria for the selection of sub-projects and a participatory approach minimize the possibility for political capture. In addition, a formal complaint mechanism appears to be an important component in mitigating risks of unfair treatment of communities or villagers.
    • Within the design of the project, a tendency toward choosing tangible construction sub-projects prevailed over soft sub-projects, such as training. It is important to promote and expand the availability and desirability of soft sub-projects that provide long-term skills and sustainability.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The evidence provided in the ICR is sufficient, and it includes both description and analysis. The report is, for the most part, internally consistent; however, its argument for the absence of appropriate labor-intensive incentives in the Second Additional Financing Credit (see, for instance, paragraph 28) is not convincing. The report makes a commendable effort to sort out the results framework. With a main text of 15 pages, the report is concise, while providing sufficient descriptive and analytical information.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions