Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Pension & Social Assistance Project

1. Project Data:   
ICR Review Date Posted:
Project Name:
Pension & Social Assistance Project
Project Costs(US $M)
 15.95  29.03
L/C Number:
Loan/Credit (US $M)
 10.00  10.93
Sector Board:
Social Protection
Cofinancing (US $M)
Board Approval Date
Closing Date
02/28/2009 02/28/2011
Compulsory pension and unemployment insurance (90%), Other social services (10%)
Social risk mitigation (33% - P) Administrative and civil service reform (33% - P) Other social protection and risk management (17% - S) Other public sector governance (17% - S)
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Robert Mark Lacey
Judyth L. Twigg Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The project development objective (PDO) according to the Project Appraisal Document (PAD, page 3) is “to significantly improve the effectiveness and transparency in the provision of public pensions and social assistance to eligible beneficiaries.”

According to the Development Credit Agreement, the PDO is “to assist the Borrower to improve the effectiveness and transparency in its public pension and social assistance systems.”

The PAD statement is more specific and monitorable in that it refers to how much improvement is sought and for whom. It is used as the benchmark for evaluation here.

b. Were the project objectives/key associated outcome targets revised during implementation?

c. Components:
There were three components:
1. Strengthening the Social Assistance Administration (US$7.35 million at appraisal, US$7.34 million at closure). This was intended to enhance the capacity of the Ministry of Labor and Social Protection of the Population (MLSPP) to deliver effectively targeted social assistance benefits to eligible families through: (i) implementation of a management information system (MIS) to support means and assets testing as a basis for eligibility to benefit from social welfare programs; (ii) implementation of targeted social welfare programs; and (iii) enhancing social protection policy analysis capabilities in the Ministry. The Component consisted of civil works, equipment, technical assistance and training.
2. Strengthening the Social Insurance Administration (US$7.58 million at appraisal, US$20.51 million at closure). This was to support the strengthening of the State Social Protection Fund (SSPF) to enable it to manage and administer the revenues and expenditures associated with the its social insurance portfolio efficiently and transparently, and to plan pension policy. It financed implementation of an MIS to support the establishment of a Central Register of individual records, and operational improvements in business processes related to the use of the Register, such as collections, monitoring of revenues and expenditures, audit inspection and management of pension client services. It also supported more general institutional enhancements (such as governance, financial and human resource management, actuarial forecasting, and public information). The Component consisted of civil works, equipment and materials, technical assistance and training.
3. Project Management, Monitoring and Evaluation (US$1.02 million at appraisal, US$1.18 million at closure). This Component supported the operations of two Project Implementation Units (PIU), one in the MLSPP and the other in the SSPF.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost. Total project cost at closure was US$29.03 million, almost double the appraisal estimate of US$15.95 million (including contingencies). Most of this increase was in Component 2 and consisted of more extensive civil works in SSPF and MLSPP offices, practically all of which were financed by the Borrower (see below).
Financing. The dollar value of the IDA credit of US$10 million, which was fully disbursed, had increased to US$10.93 million at closure, due to depreciation of the US dollar against the SDR. There was no external parallel financing or cofinancing.
Borrower contribution. This was US$18.10 million at closure, compared to an appraisal estimate of US$5.95 million. It included a UNDP grant (the amount of which is not specified in the ICR, but which the project team subsequently informed IEG to be US$800,000) which the Borrower had obtained prior to the project to bridge the time gap before the project became effective. According to the project team, a further US$1.33 million was contributed by UNDP during implementation. The activities financed by these grants and from the Borrower’s own resources were effectively coordinated with the activities carried out under the project. Much of the Borrower’s contribution (including the grant) was used to rehabilitate MLSPP and SSPF offices to accommodate the MIS at both central and regional levels. This freed up IDA resources, which were originally intended to cover civil works, for development of the MIS itself. In the case of MLSPP, the MIS covered all offices and programs at both central and local levels by project closure, although it was initially foreseen that it would cover only the Targeted Social Assistance program.
Dates. The closing date was extended twice for a total of 24 months. The first extension (granted on September 17, 2008) was for one year to February 28, 2010, in order to complete analysis of the TSA program survey, procure and install servers and other IT equipment in SSPF centers, and complete works related to the installation of the MIS for SSPF. The second extension (granted on January 15, 2010), until February 28 2011, was to complete the bidding process for the MIS for MLSPP, to complete procurement processes for IT equipment for both implementing agencies, and to conduct a survey of the impact of pension reforms supported by the project.

3. Relevance of Objectives & Design:

a. Relevance of Objectives:
The project’s goals are substantially relevant to the Bank’s 2011-2014 Country Partnership Strategy (CPS) for Azerbaijan, current at the time of project closure. The CPS describes the progress made under the previous CPS towards a system of more modernized, transparent and well-targeted social assistance as one of the more encouraging areas of IDA’s cooperation with Azerbaijan. Much of this effort was supported by the project. The CPS advocates continued improvements in order to make a further dent on poverty and raise incomes. This would be through consolidation of the reforms already achieved combined with a sharper focus on reaching the very poor. The objectives were also relevant to the previous (2007-2010) CPS, the priorities of which included enhanced social protection of vulnerable groups (including pensioners, children and the unemployed) and improved access to public services.
The PDO is also substantially relevant to the Government of Azerbaijan’s comprehensive social assistance reform strategy aimed at increasing the effectiveness, efficiency and transparency of its social protection system. According to the State Program for Poverty Reduction and Economic Development, this system is a crucial tool for addressing poverty and creating the conditions for faster growth of employment and incomes. The Program identified a number of issues hindering the efficient and effective use of social insurance and assistance resources, including low benefits and coverage, complexity of benefits, a weak link between contributions and benefits, poor record keeping, opaque and undeveloped financial management systems, poor targeting of social assistance beneficiaries, weak monitoring, evaluation and policy planning capacities, and poor capacity to administer benefits.

b. Relevance of Design:
The PDO was clearly stated and the Results Framework adequately represented the underlying reasoning linking the project’s inputs and outputs to the outcomes expressed in the PDO. In the Results Framework (see page 17 of the PAD), the activities financed by the project feed logically and realistically into outputs and intermediate indicators, which, in turn, could be expected to support attainment of the PDO through achievement of the main outcome indicators. For example, Component 1 financed the development and implementation of an MIS (including a good targeting mechanism) for MLSPP whose staff are trained to use it, and which could be expected to achieve the following outputs and intermediate outcomes: (i) a more transparent and efficient means of testing eligibility; (ii) a reduction in targeting errors; and (iii) shared information between agencies and offices to promote better targeting. These, in turn, could be expected to lead to achievement of the PDO.

4. Achievement of Objectives (Efficacy) :

To significantly improve effectiveness and transparency in the provision of public pensions to eligible beneficiaries. Substantial
  • Technical advice in the preparation of new pension legislation, approved over the period 2001-2005, which introduced, inter alia, the creation of individual pension accounts and the removal of ceilings on contributions.
  • Establishment of an interconnected MIS system in all SSPF offices, both central and local.
  • Establishment of a Central Registry, which allows a complete and updated record of participants.
  • Modernization and streamlining of collection and payment processes in SSPF. The entity now uses a fully automated, individualized record keeping and benefit administration and management system, which has “strengthened administrative discipline” (ICR, page 14). Pensioners are paid by through VISA debit cards which can be used at ATM machines.
  • A public awareness campaign conducted by SSPF to inform its clients and the public more widely about the reforms and their expected impact.
  • By project completion, nearly all social insurance beneficiaries were receiving their pensions on time and without arrears. Neither the ICR nor the PAD indicates the extent of arrears prior to the project, but the ICR describes them (Data Sheet, page iii) as “significant.” The ICR reports (page 14) that the virtual elimination of arrears was achieved through the establishment of the Central Registry and by the automation of the payment process, both supported by the project. These have greatly reduced delays in the delivery of pension payments.
  • The project has contributed significantly to greater financial viability of the social insurance system. As the following table shows, contribution collection increased fivefold between 2003 and 2010 (the PAD states on page 45 that “the Project aims at [an] 80 percent increase in contributions collection in 2009;” the actual increase was 564%). Over the same period, annual contributions collected per participant rose from ANZ172 (US$221) to ANZ651 (US$835). These improvements reflect both a higher compliance rate and a 45% increase in the number of contributors. They are considerably greater than could be explained by the relatively favorable macroeconomic environment that prevailed until 2008. Indeed, both total contributions and numbers of contributors continued to increase after 2008, despite a much less conducive economic background.
  • The project has also contributed to the pension system’s fiscal sustainability. The percentage of total SSPF revenues represented by transfers from the government budget showed a declining trend over the period 2003-2010 as a whole, although increases were recorded from 2008 onwards (see the table below). These latter reflect the impact of the deteriorating macroeconomic situation caused by the international financial crisis. However, given the importance of the project in securing higher contributions from a much larger number of participants, it is reasonable to assume that, without the project, budget transfers to the SSPF during the crisis years (2008-2010) would have been significantly higher both in volume and as a percentage of total revenues.
YearSSPF revenues from contributions (ANZ millions)SSPF transfers from the Budget (ANZ millions)Total SSPF revenues
(ANZ millions
Percentage of Budget contributionNumber of contributors (millions)
2003223154 37740.81.30
2004279137 41632.9N.A.
2005316186 50237.1N.A.
2006423167 59028.3N.A.
2011N.A.N.A.N.A.N.A.1.88 (January)
Source: ICR, pages 13 and 14.
  • A beneficiary survey conducted in 2010 by SSPF demonstrated a higher degree of client satisfaction with the social insurance system than did a similar exercise carried out during preparation. However, the results show that considerable scope remains for further increases in satisfaction. The results, cited by the Borrower's ICR (ICR, page 43), are: (i) 33% of respondents mentioned the increased amount of the pension and its timely payment as the main achievement of the reform, (ii) 52% linked more timely payments to the introduction of ATM cards, and (iii) 11% considered that computerized administration of pension calculation mechanisms contributed to improved services. The ICR (page 10) points out that, the results reflect a continuing need to raise public awareness campaign on need for, and benefits accruing from, pension and social assistance reforms.
  • Project activities contributed to the increased transparency of the public pension system in a variety of ways. The approval of the Pension Law at the end of 2005 set the ground for information sharing with beneficiaries both about their own pension rights and expectations through creation of individual accounts and through establishing a solid data base of individual records. The legislation also permitted more general information about the pension administration and reform process through an active public information campaign. Online account access for beneficiaries was to be permitted from 2011 onwards. The establishment of the Management Information System and accompanying technology automating both information flows and payments procedures reduced the opportunities for fraud and corruption, and increased the accountability of SSPF staff. The fact that, by project closure, all recipients were receiving their full pension payments on time also contributed to enhanced transparency since arrears can easily add to confusion about how much is due. The greatly improved record keeping associated with the Central Registry permits accurate and up-to-date monitoring of whom has been paid how much and when.
To significantly improve effectiveness and transparency in the provision of social assistance to eligible beneficiaries. Substantial
  • In 2006, with project support, Azerbaijan introduced Targeted Social Assistance (TSA), which consolidates the state-provided basic pension for the uninsured as well as several child and family benefits, thereby decreasing fragmentation and potentially securing important efficiency gains.
  • The launching of the TSA was preceded by the adoption of several key pieces of legislation and related regulations, the preparation of which received technical support from the project.
  • Due to support from the project, MLSPP now has a fully automated TSA administration with real time connection covering the entire country. The majority of TSA beneficiaries receive their monthly benefits by using an ATM card. MLSPP also has an MIS in place which includes a means and asset testing targeting mechanism based on a successful EU-supported (TACIS) pilot program implemented in 2003. Comprehensive computerized record-keeping of eligible households and payments was set up in all social welfare offices. A TSA centralized database program was also established, providing comprehensive information on the TSA recipients, applicants and resources spent on the program. Records are updated in real time.
  • By project completion, all beneficiaries were receiving their social assistance benefits in a timely manner and without arrears. Arrears prior to the project were described as “significant.”
  • The elimination of benefit arrears was, however, facilitated by the favorable macroeconomic and fiscal environment which prevailed until 2008. When budgetary pressures reappeared following the international recession of 2008-2009, TSA pensions “briefly” fell back into arrears (ICR, page 7).
  • Support from the project has also strengthened the poverty focus of social protection thanks to better targeting. According to the PAD (page 46), the 2003 EU-supported pilot showed the targeting error for social assistance payments to be 50% -- i.e. half of the recipients were not eligible according to the established income and asset criteria. The project target was to bring this down to no more than 40%. A 2008 survey of the TSA showed the error to be 30.6%. Although this is still high, it is a considerable improvement on what obtained before the project. More recent data from a 2011 Living Standards Measurement Survey (LSMS) were not available when the ICR was completed. However, the ICR reports (page 17) that, in 2010, 49% of TSA beneficiaries were from the lowest income decile of the population, and 86% from the lowest 20% (no comparable figures are available from the period prior to the project).
  • The ICR quotes a 2008 LSMS which showed that, without social transfers, the overall poverty incidence would increase from 10.8% to 21%, and the poverty gap would more than double (from 2.4% to 7.2%). The study also indicates that most of the poverty reduction effect of social transfers has been due to retiree social assistance benefits (not the social insurance pensions administered by the SSPF), which comprise 85% of the total transfers made under all social protection programs in Azerbaijan. According to the study, for each percentage point of GDP spent on transfer to older age beneficiaries and other retirees, poverty declines by 2 percentage points; for the TSA program the ratio is 1:2.8. However, the extent to which this can be attributed to the project is difficult to determine since increases in the per capita benefit have been an important factor in raising the incomes of poor households. Between January 2005 and January 2008, the minimum pension paid to all beneficiaries rose almost 2.5 times from AZN25 (about US$32 to AZN60 (US$77), and between 2006 and 2011, the average TSA benefit almost doubled from AZN44 (US$56) to AZN100 (US$128).
  • Project activities have also been conducive to enhanced transparency in the delivery of social benefits. The automated payment system has reduced the opportunities for fraud and corruption, and increased the accountability of MLSPP staff. The elimination of delays has also contributed to transparency by removing sources of confusion for beneficiaries. The establishment of clear criteria for social assistance eligibility and more accurate methods of asset and means testing enhances the system's transparency as well as the quality of its targeting.

5. Efficiency:

There is no economic analysis or any attempt to assess the efficiency of the project in other ways either in the PAD or in the ICR. The PAD estimates the project’s future impact on SSPF revenues. The ICR (page 16) makes a qualitative statement to the effect that the project would be conducive to efficiencies due to better administration, targeting of payments, reductions in allocation errors and consolidation. However, even if an attempt had been made to quantify them, these are efficiencies pertaining to the sector and do not measure efficiency in the use of project resources. For lack of information and analysis, efficiency must be rated modest. The fact that total project cost was almost double the appraisal estimate (though some of this is accounted for by additional activities), and the two year time overrun are also indications of modest efficiency.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:

Rate Available?
Point Value
ICR estimate:

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The project substantially fulfilled its objectives of improving effectiveness and transparency in the provision of both public social insurance pensions and social assistance. Project interventions supported enhanced levels of service (timely and accurate payments), improved financial and fiscal viability, better targeted and more poverty-focused social assistance benefits, and an increase in client satisfaction. Relevance of both objectives and design was also substantial. Efficiency, however, has to be rated modest due to lack of information and analysis. Taken together these rates constitute moderate shortcomings, and therefore overall outcome is assessed as moderately satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

  • The technical risk to development outcome is low. Indeed, one of the features of this project is that training of staff of both SSPF and MLSPP in the use of the equipment, systems and processes installed under the project was given the utmost priority (ICR, page 8). This training-oriented culture continues to prevail. If private pension funds and insurance were to develop, then it is likely that the state system would lose human resources, but they would remain at the disposal of the sector.
  • The main medium term financial risk is macroeconomic. Like social protection schemes everywhere, contributions and outlays vary according to the levels of employment, wages and economic growth. Nevertheless, the reforms supported by the project have been shown to be robust under macroeconomic stress.
  • Both social insurance and benefit transfer programs in Azerbaijan are dependent on budget transfers constituting a long term risk. However, according to the ICR (page 18), the Government has shown strong interest in enhancing actuarial analysis with a view to addressing long term fiscal sustainability of the social insurance program.
  • The social risk is considered low. Clients have expressed high levels of satisfaction with the improvements, and there is a greater buy-in to social insurance from previously under-represented groups such as farmers and the self-employed.
  • The political risk is minimal. Government ownership and commitment to sustaining and deepening the gains stemming from the project is strong as demonstrated, inter alia, by their request for a follow-up IDA supported Social Assistance Development Project (now ongoing).
  • Stakeholder ownership is growing. Not only is there an increased level of client satisfaction, but the ICR reports (page 18) the development of a more client-and-efficiency oriented culture among the staff of both SSPF and MLSPP.

    a. Risk to Development Outcome Rating: Negligible to Low

  • 8. Assessment of Bank Performance:

    a. Quality at entry:

  • Given the Government’s decision to strengthen the poverty-orientation of social protection programs and to enhance the financial and fiscal viability of the social insurance program, the preparation team’s emphasis on the administrative and technical means to achieve these ends was appropriate. The project was a logical continuation of previous IDA analytical and advisory assistance and policy dialogue in the sector.
  • The technical quality of the background analysis preceding and during preparation was sound. It provided the basis for the advice tendered by IDA for improved pension related legislation and regulations. Project preparation was informed by a PHRD grant which financed technical assistance to the Government in designing the operation, and by a World Bank Poverty Assessment which provided important insights. Preparation also benefited from: (i) lessons learned from the experience of social protection and labor market components of previous operations in Azerbaijan, including Development Policy Credits (for example, the importance of government ownership and active involvement by Ministries and agencies in project implementation, building a broad consensus for change, realistic and flexible implementation timetables, and a robust M&E system); (ii) Bank experience in helping to develop social insurance and social assistance administration in other countries; (iii) analytical work; and (iv) complementarities with European Union financed monitoring of a pilot means-testing program.
  • The Bank’s international experience and analytical capabilities enabled the preparation team to play an important catalytic role in building consensus for social protection reform and in overcoming resistance within the affected institutions.
  • The high priority given in project design to training of SSPF and MLSPP staff was to pay important dividends later, since both institutions were able to use the project to upgrade their existing staff and make strategic hirings.
  • Most risks were appropriately identified: (i) delays caused by limited implementation capacity; (ii) reluctance to use external technical assistance for capacity building; (iii) inability of implementing agencies and Project Implementation Units (PIUs) to retain key staff; and (iv) weakening of government commitment to continue with system reforms. The absence of the macroeconomic risk is, however, noteworthy; indeed, macroeconomic dimensions receive little attention in the PAD. Ratings and mitigation measures for the risks identified were generally adequate. However, the ICR notes (page 7) that risk (i) “could have been rated as High instead of Medium. From the outset it could have been expected that the design and implementation of information systems with lengthy procurement processes for hardware and software were likely to take longer than estimated at appraisal. In practice, lengthy government processes with regard to procurement necessitated extension of the closing date for a total of 24 months.”
  • The Results Framework design was adequate, though there were some gaps in baseline indicators and targets (see Section 10 below).
  • The main shortcoming was the lack of an adequate analysis and discussion of ex ante efficiency in the PAD (see Section 5 above).

  • Quality-at-Entry Rating: Moderately Satisfactory

    b. Quality of supervision:

  • The project was regularly supervised with eleven formal missions taking place during the six-and-a-half years between approval and closure.
  • All Implementation Status Reports recorded both progress toward PDO achievement and in implementation as fully satisfactory throughout. This demonstrates a lack of candor with regard to implementation. The ICR notes that the procurement-related delays led to two 12 month extensions of the closing date. These are not reflected in the ISR ratings.
  • There was continuity in team leadership, with only one change since the inception of preparation. The first TTL, who had guided the project through approval, provided implementation support until the end of 2009. The second TTL had been a member of the supervision team, thus facilitating a smooth transition without adverse consequences. The ICR reports (page 19) that the skills mix of supervision teams was appropriate. They included Country Office staff which ensured timely follow-up on issues between formal missions.
  • The Borrower’s ICR (ICR, page 39) expressed satisfaction with the role of the IDA team which “worked closely with the Borrower to support its efforts to successfully implement the project. The dialogue during the implementation process was smooth and both parties sought and found solutions to implementation bottlenecks (civil works procurement and financial aspects). The World Bank supervised the project regularly and this intensive supervision was one of the major contributing factors in achieving the targets.”
  • The supervision team showed flexibility and pro-activity in making amendments to the DCA in response to changing circumstances.
  • The ICR notes (page 20) that field visits were regularly made to regional and local offices of SSPF and MLSPP, and that these were particularly appreciated as opportunities to exchange information and to ensure that local needs and viewpoints were taken into account.
  • The team worked closely with the PIUs to ensure the satisfactory implementation of M&E, which was facilitated by the installation of the two MIS and related staff training.
  • Although SSPF had signaled its intention to use Borrower resources for civil works, it was not until the second semester of 2009 that the reallocation of the amounts in the IDA credit originally destined for civil works was made. This led to a disbursement flag being raised (see Section 11 below).
  • Efficiency in the use of project resources does not appear to have been tracked during implementation. The total project cost was almost double the appraisal estimate (though some of this is accounted for by additional activities) and there was a two year time overrun both of which raise some question about the efficient use of project resources.

  • Quality of Supervision Rating: Moderately Satisfactory

    Overall Bank Performance Rating: Moderately Satisfactory

    9. Assessment of Borrower Performance:

    a. Government Performance:

  • The Government showed strong commitment throughout preparation and implementation as demonstrated by its efforts in getting key legislation and regulations approved. The Pension Law of late 2005 established the basis for defining and streamlining the new pension system and for the utilization of individual records. Without the Law, SSPF would have been unable to proceed with project implementation. The Law also set the ground for information sharing with beneficiaries, thereby improving transparency and governance.
  • Similarly, a number of regulatory acts stipulating the organization and implementation of the TSA were developed and adopted by MLSPP, including detailed instructions on how to implement the targeting mechanism.
  • Although the Minister of Labor changed in 2005, the new Minister demonstrated his engagement in the project from the outset. According to the ICR (page 8), “his personal involvement was a critical contributing factor to [the project’s] successful implementation.”
  • The Government’s counterpart contribution, including a grant it had obtained from UNDP prior to the project, was over three times what had been foreseen at appraisal and enabled productive reallocation of the IDA credit (see Section 2d above).
  • Prior to, or during, preparation, the Government made the following important contributions to the project: (i) development of key strategic documents such as the State Program for Poverty Reduction and Economic Development, which envisaged a comprehensive reform of the social protection system; (ii) issuance of a Presidential Decree in 2001 separating social insurance from social assistance benefits administration; (iii) adoption, also in 2001, of a Pension Concept to modernize the social insurance system pursuing market economy principles; and (iv) development of a means testing targeting mechanism on the basis of a pilot program implemented in 2003 with support from the European Union TACIS.
  • The main shortcoming in government performance concerned the slow and cumbersome review and approval process of both procurement packages and disbursement documentation, which required the participation of the Ministry of Finance, the Ministry of Economic Development, and, in some instances, the entire Cabinet. This caused considerable implementation delays. The ICR reports (page 19) that procedures were streamlined after the Mid-Term Review in December, 2007.

  • Government Performance Rating: Satisfactory

    b. Implementing Agency Performance:

  • The ICR reports (page 21) that the management teams of both implementing agencies (SSPF and MLSPP) were committed to the project. The heads of both agencies involved themselves systematically and continually in following project progress.
  • The institutional arrangement of each implementing agency having its own PIU responsible for project administration worked well. The directors and staff of the PIUs, for the most part, remained in their position for the duration of the Project. The PIUs worked in close collaboration with the technical teams in each agency. The two PIUs remain in place and are performing the same role under the follow-up Social Protection Development Project.
  • The ICR reports (page 21) that the PIUs played a valuable coordinating role between the two agencies, and among central, regional, and local offices to ensure timely implementation of activities. They also liaised well with the IDA team bringing issues that arose to its attention in a timely manner, and displayed initiative in fine-tuning programs in response to feedback from clients in surveys conducted throughout implementation.

  • Implementing Agency Performance Rating: Satisfactory

    Overall Borrower Performance Rating: Satisfactory

    10. M&E Design, Implementation, & Utilization:

    a. M&E Design:
    The Results Framework presented in Annex 3 of the PAD was, in principle, adequate for monitoring and evaluating the project. There were four main outcome indicators – (i) improved financial viability of the social insurance system (higher collections); (ii) significant increases in the number of participating contributors with a compliance rate of 90%; (iii) eligible beneficiaries for both social insurance and social assistance receive full payments in a timely manner; and (iv) improved user satisfaction – and 12 intermediate indicators, which together provided a logical linkage between the activities supported by the project and the intended outcomes. However, several of the indicators lacked baseline values and targets, with the latter being added during implementation. The project’s intention was to support the installation and operation of MIS in both implementing agencies which was expected, in and of itself, to lead to a robust M&E system. Regular surveys were planned to capture client reaction to the changes carried out. Responsibility for M&E rested with the PIUs.

    b. M&E Implementation:
    The ICR reports (page 9) that M&E functions were carried out satisfactorily over the life of the project by the PIUs. Reliable data were made available in a timely manner. The MIS was installed in both implementing agencies, leading to quicker availability of higher quality information. The Results Framework was refined and updated throughout implementation to add clarity to the language and to ensure that results could be measured. Client satisfaction surveys were carried out as planned.

    a. M&E Utilization:
    The project’s M&E system has served to inform policy making by enabling the continuing assessment of efficiency, effectiveness, and transparency in the provision of pensions and social assistance. Both central and local authorities are being provided with timely and relevant data which they are using to adjust the programs. For example, on the basis of the results of the 2008 survey, the TSA was fine-tuned and amendments introduced to the TSA Law that made the application and administration processes easier for beneficiaries and staff responsible for its administration. M&E has become an important management tool in both implementing agencies.

    M&E Quality Rating: Substantial

    11. Other Issues:

    a. Safeguards:
    Although this was a Category “C” project for purposes of environmental safeguards, and no safeguards policies were triggered, attention was paid during preparation to the adequacy of hazardous material treatment in office renovation. The ICR states (page 10) that “provisions were included in civil works contracts with regard to disposal of materials in an environmentally safe manner, and according to the PIUs, compliance was monitored regularly by the Engineer/Architect in the PIUs during implementation of the works and found to be satisfactory.”

    b. Fiduciary Compliance:
    Financial management. According to the ICR (page 11), financial management arrangements at the PIUs, including internal controls and accounting, reporting, flow of funds and staffing, were considered to be adequate and were consistently rated as satisfactory in the ISRs. As noted, the Government exceeded its counterpart financing obligations, and its contributions were made in a timely manner. Audits. The ICR reports that external project audits were prepared and submitted in a timely manner and were consistently unqualified.
    Procurement. At the level of the implementing agencies, procurement was managed in a generally satisfactory manner. There were procurement specialists in the PIUs who received training in IDA-organized seminars and workshops. The ICR (page 11) reports that procurement planning and implementation were satisfactory, despite the necessity of complying with both Bank and internal government guidelines. As noted in Section 9 above, the main challenges arose from the need for high government level approval for any changes in the procurement plan. These issues were partially overcome following the Mid-Term Review.
    Disbursements. A disbursement flag was raised in the second semester of 2009 because the decision not to use IDA credit resources for most civil works meant that funds allocated to this under Component 2 remained unspent. After the reallocation to expenditures on the two MIS, all IDA funds had been disbursed by closure.

    c. Unintended Impacts (positive or negative):
    None reported

    d. Other:

    12. Ratings:

    IEG Review
    Reason for Disagreement/Comments
    Moderately Satisfactory
    Efficiency must be rated modest due to the inadequacy of information and analysis. The total project cost was almost double the appraisal estimate (though some of this is accounted for by additional activities) and there was a two year time overrun.  
    Risk to Development Outcome:
    Negligible to Low
    Negligible to Low
    Bank Performance:
    Moderately Satisfactory
    There was little attempt to analyze the efficiency of the project. 
    Borrower Performance:
    Quality of ICR:
    - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
    - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

    13. Lessons:
    The following are the most important lessons from the preparation and implementation experience of this project. The first is drawn by IEG. The others are taken from the ICR with some adaptation:
    • Installation of MIS and similar new technology needs to be accompanied by a thorough training program in its use for the staff of the entities concerned. In this case, the emphasis placed on such training from project preparation onwards, and the inculcation of a training culture in the implementing agencies, paid significant dividends.
    • Stakeholder acceptance of reform and system improvements, even when they result in enhanced client services and are accompanied by higher benefits, should not be taken for granted. This project included a systematic public awareness campaign. Despite this consistent effort, surveys conducted towards the close of the project indicated a need for a continuation and deepening of the campaign.
    • Reforms in the social assistance sector require a long term engagement. In this case, IDA has succeeded in building a position of trust as a strong partner who is there for the long haul. This project was preceded by intense policy dialogue and social assistance sector interventions in other operations, including Development Policy Credits. It is followed by another ongoing project to sustain and expand the reforms.
    • Improving governance and accountability are central elements to a successful reform of social protection programs. In this case, both the social insurance and social assistance systems prior to the project were characterized by lack of transparency in eligibility criteria and opportunities for corruption. The client surveys indicate that, while much has been done, building public trust is still an ongoing task.

    14. Assessment Recommended?


    15. Comments on Quality of ICR:

    The ICR is clearly written and its evaluation of project outcomes is evidence-based. However, the inadequacy of the efficiency discussion, which focuses on outcomes rather than on the efficiency with which IDA resources were used, is a shortcoming. An explanation of why the IDA funds destined for civil works were only reallocated in the second semester of 2009 would have been useful. The contribution of the UNDP grant to counterpart funding is not quantified.

    a. Quality of ICR Rating: Satisfactory

    © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions