Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Social Sector Reform Dpl


  
1. Project Data:   
ICR Review Date Posted:
10/07/2013   
Country:
Albania
Is this review for a Programmatic Series?
 No
First Project ID:
P116937
Appraisal
Actual
Project Name:
Social Sector Reform Dpl
Project Costs(US $M)
 25.0  25.0
L/C Number:
Loan/Credit (US $M)
 25.0  25.0
Sector Board:
Social Protection
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  04/28/0211
 
 
Closing Date
04/30/2012 04/30/2012
Sector(s):
Other social services (50%), Health (50%)
Theme(s):
Social safety nets (40% - P) Health system performance (40% - P) Other social protection and risk management (20% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Hjalte S. A. Sederlof
Robert Mark Lacey Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Program Document (page 26), the Project Development Objective (PDO) was to support policy changes to: (i) improve the effectiveness of social safety nets, and (ii) enhance the efficiency and equity of health spending, in a fiscally sustainable environment.

    As this was a Development Policy Operation, there was no statement of objectives in the legal documents.

    The Program Document (page 2) included the following outcome targets:
      • increased percentage of the poor receiving means-based social assistance (22 percent in 2008; target 30 percent in 2012);
      • savings realized in disability benefit spending of at least 5 percent of its budget by 2012;
      • percentage of payments to hospitals channeled through the health Insurance Institute remains constant or increases (by 2012)
      • health spending as a share of household spending for the poor decreases by at least 5 percent.

b. If this is a single DPL operation (not part of a series), were the project objectives/key associated outcome targets revised during implementation?
No

c. Policy Areas:
1. To improve the effectiveness of social safety nets, the Operation would support policies that:

    1. strengthen the mechanisms to allocate funds and select beneficiaries in the main social assistance program;
    2. introduce incentives for investing in human capital by providing an additional benefit linked to school enrollment and attendance for families with school age children;
    3. initiate the process to create a unified registry of beneficiaries of social assistance programs; and
    4. change the indexation formula of the disability benefits program, to control the share of resources allocated to this program, which is growing disproportionately;
2. To enhance the efficiency and equity of health spending, the Operation would support policies to
    1. introduce more predictable fiscal rules to finance health;
    2. improve the methods for contracting with providers and the capacity to monitor their performance; and
    3. grant health insurance coverage to social assistance beneficiaries.
It would also enhance transparency and accountability by supporting policies to improve the administration of social assistance programs and promoting the use of systematic formulas to allocate resources in selected social programs and services.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Cost and Financing. The original commitment to the DPL was a single-tranche US$25 million Loan. It was fully disbursed.

Borrower Contribution: None

Dates: The project was approved on April 28, 2011; it became effective on August 24, 2011 and was disbursed on August 25, 2011. The Closing Date for the Project April 30, 2012.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Relevance of objectives is rated high.

Project objectives were to support policy changes in safety nets and health spending while ensuring fiscal sustainability - in the short run through policy measures, and in the longer term through higher efficiency and effectiveness in resource use in the two target areas. This was in line with government objectives as expressed in the 2007-2013 strategy for national development and in the Bank’s Country Partnership Strategy for 2011-2014. Both emphasized rapid, balanced and sustainable economic, human and social development, including such features as higher school enrollments, increased access to a basic package of health services and better coverage by social assistance, all in a fiscally sustainable manner.

b. Relevance of Design:

Relevance of design is rated modest.

The policies supported by the DPL and reflected in the prior actions, were partly aligned with the PDO. However, the policy actions included in the design, the lending instrument and the time frame were insufficient to achieve the objective. While the policy matrix (PAD, pp. 52-53) defined links between prior action-related outputs and outcomes, the causal chain between the indicators and the PDO was less well established. In addition, several key outcome indicators were not achievable under the project, and should have been designed as long-term goals rather than Project outcome indicators.

In safety nets, increased coverage of the poor can serve as an indicator of increased effectiveness, but reduced spending on disability benefits says little about effectiveness. The region subsequently clarified that Bank analytic work has clearly established that the uncontrolled growth of poorly targeted disability benefits was squeezing out spending on targeted programs for the poor, and necessary to improving effectiveness of these programs. In health, channeling hospital payments through the Health Insurance Institute, or maintaining them at previous levels are not per se indicators of either efficiency or equity in health spending; while decreased spending by the poor (as an equity indicator) is a relevant equity indicator only when it is analyzed in the context of health spending across income groups and together with the level of utilization of care. A school attendance bonus is included as part of the safety net design and as a prior action. However, neither enrollment nor attendance are listed as major issues in the Bank's 2010 Economic Report on Albania, nor does the Government's reluctance to proceed on this issue beyond fulfilling the prior action indicate particular urgency.

Project design also raises additional concerns:
A DPL was chosen to ease possible short-term fiscal constraints on expanding coverage and raising quality of social assistance and health programs. However, it is not immediately obvious that a DPL was necessary. (i) The Government already was committed to reform of social assistance and health. (ii) Ongoing social sector investment lending already provided coordinated vehicles for providing necessary investment and technical assistance; and a further social assistance project was under preparation (P055383, P082814, P122233). According to the ICR (page 33), measures being introduced under those projects are not expected to have a significant adverse fiscal impact. (iii) The general fiscal situation was tight but not alarming. Albania was one of the few countries in Europe to have maintained positive growth throughout the past decade, including during the global downturn in 2009. Although Albania did come out of that crisis with a worsened fiscal balance, at the time that the project was being considered, the fiscal situation had already started to improve and the budget deficit showed a declining profile over the short- to medium term. Moreover, the country had low and falling poverty rates that were projected to continue falling; and spending on social assistance did not appear unduly low or out of line with comparable countries.


4. Achievement of Objectives (Efficacy) :

All prior actions for the release of the DPL were met. Efficacy is assessed on the basis of three objectives. Given ongoing investment projects and the involvement of other donors in the two sectors, many of the outcomes indicated below were, or are likely to be, achieved through a combination of legislation supported by the DPL, Bank investment projects and other donor support. The DPL supported policy actions that were to achieve three sub-objectives.

Objective 1: to improve the effectiveness of social safety nets

Efficacy is rated modest.

Outputs
A social assistance framework law was adopted, revising eligibility criteria; and the resulting revised targeting mechanism was piloted;
The process to establish a unified registry for social assistance beneficiaries was launched
The disability benefit formula was changed to link adjustments to price inflation (instead of wage inflation)
Legislation on incentives for school attendance was adopted

Outcomes
Savings in the disability benefit program of 7 percent between end-2008 and end-20011, against a target of 5 percent. The target was surpassed.

The improved targeting mechanism and the unified registry were to result in an increase in (i) the share of the poor receiving means-tested social assistance from 22 percent in 2008 to 30 percent in 2012; and (ii) the share of benefits accruing to the poor from 42 percent in 2008 to 60 percent in 20013. Based on available information in the ICR that draws on monitoring results from the Ministry of Labor and estimates by the project team, the share of the poor receiving means-tested social assistance will exceed 30 percent in 2013. Based on the same sources, the target for benefits accruing to the poor also will be met with a delay.

The last outcome target - bonuses for school attendance - was not achieved. The region further explained that school enrollment and attendance rates are much lower among social assistance (Ndhima Ekonomika) recipients and particularly among the Roma. The planned school attendance bonus in Ndhima Ekonomika targeted these groups directly. An accompanying outcome indicator (not a key indicator) had been set whereby attendance by children from social assistance beneficiary families was to increase from 52 percent in 2010 to 70 percent by 2013; no target was set for the project period. However, government initially indicated commitment to pursuing this approach, but it was not implemented

Objective 2: to enhance the efficiency of health spending

Efficacy is rated modest.

Outputs
Performance-based single-payer purchasing of hospital services through the Health Insurance Institute (HII) was introduced;
A systematic transfer of resources from the state budget to the Health Insurance Institute was introduced;
The new Health Insurance Law was passed by the Parliament in 2012.

Outcomes
The new insurance and provider payment arrangements have been adopted nationwide: the HII has signed performance contracts with 39 hospitals; and 87 percent of payments to hospitals made through the HII
There has been a 20 percent reduction in hospital lengths of stay during the last two years; however, there is no information on hospital occupancy rates, and other productivity indicators such as the number of staff per occupied bed;
A 10 percent increase in the volume of work in 2012 in Albania's one tertiary hospital with a budget increase of 3 percent. The ICR does not explain what the "volume of work" encompasses.

No information is provided on how these measures have affected the efficiency (cost-effectiveness) of health spending; however, it could be expected that the provider payment reforms supported under this loan will contribute to more efficient health spending.


Objective 3: to enhance equity of health spending

Efficacy is rated negligible

Outputs
Health insurance with a defined benefit package was extended to all social assistance beneficiaries

Outcomes
It is not clear how this output affected equity, unless there is a baseline, for instance some clarification of how social assistance beneficiaries paid for their health care prior to the new arrangements.

The outcome indicator – a decline in health spending as a share of household spending for the poor decreases by at least 5 percent – was met. However, this is not a sufficient indicator. In order to assess equity, health spending needs to be looked at as a share of household spending across all socio-economic groups, and combined with utilization. The non-poor may pay a higher share because they go to more expensive private sector providers; the poor might spend 5 percent less because they don’t use care at all.


5. Efficiency (not applicable to DPLs):

6. Outcome:


Based on the high relevance of the PDO; the modest relevance of design; the modest achievement of the first and second objective and negligible achievement of the third objective; the operation’s outcome is rated unsatisfactory. While the operation was responsive to country conditions and the Bank’s overall strategy, and drew on other Bank operations in the two sectors, its design did not correspond well to the PDO. While the operation was responsive to country conditions and the Bank’s overall strategy, and drew on other Bank operations in the two sectors, its design did not correspond well to the PDO. The choice of lending instrument and implementation time frame were inadequate for achieving the objectives. The results framework would offer little guidance on what improvements in outcomes could be expected, and in particular how the outcome indicators chosen would meet the PDO. The project was able to point to few results: it provided only limited evidence of achievement on most key outcome indicators. Finally, it provided no assessment of the fiscal sustainability of the set of expanding programs it was supporting.


a. Outcome Rating: Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:


The main risks that were identified at appraisal were macro-economic, including fiscal ones, and political and technical risks. All appeared moderate at the time, and they remain moderate to low. The country continues to pursue a conservative fiscal policy, and any negative fiscal effects of reforms to the safety net and to health spending are expected to be minor. While the development outcomes of the DPL are rated unsatisfactory, the situation may change over time, as the measures underpinning the DPL (and developed under the related investment projects) are implemented and begin to show results. The Government backs the continued implementation of the measures taken under the DPL, and this is reinforced by the presence of outside support provided by a follow-up Bank project and other donors active in safety nets and health.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:
The DPL was able to draw on extensive prior analytical work and technical assistance in social protection and health. The preparation and implementation of two health projects and a health sector note, framed by an ongoing dialogue on health sector policy reform, were available to draw on for the health part of the DPL. The social protection part of the DPL was underpinned by extensive technical support by means of a programmatic multi-year technical assistance program in social protection, including workshops, diagnostic assessments, reports and policy notes, as well as active dialogue leading up to the DPL. However, the Bank team did not sufficiently appraise the operation's design so that it would achieve the planned development outcomes. The choice of lending instrument and implementation time frame were inadequate for achieving the objectives. The results framework would offer little guidance on what improvements in outcomes could be expected, and in particular how the outcome indicators chosen would meet the PDO (Sections 3b, 4 and 6).

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
Supervision consisted of two missions for the one-year DPL. This was complemented by Headquarters and Country Office staff working on related projects. The ICR reports that “interaction with [related] operations…has been mutually reinforcing” and “transition to the follow-up operations has been carefully planned and has resulted in a smooth transition”. (see ICR page 14 (Figure 1)). The supervision team did not focus sufficiently on the development impact of the project, that is, the relationship between outcomes and actions supported by the DPL: and neither the outcome indicators nor the PDO were adjusted during implementation, despite the discontinuity in the results chain and the missed indicator targets.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
Government commitment to achieving the development objectives appears high when placed into the programmatic context of social sector reform in Albania, illustrated on page 14 of the ICR (Figure 1). It was able to mitigate risks identified at appraisal by maintaining prudent fiscal policy, achieved broad support for reforms across political divisions, and coordinated reform actions across sectors. However, the DPL experienced slower than expected implementation and an absence of information to facilitate a systematic monitoring of commitments under the DPL, which in turn contributed to the fact that key targets were notmet. And the postponement of the education component indicates an initial over-estimation of capacity among local entities to introduce it in the time-frame that originally was set under the DPL.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:
Not Applicable.

Implementing Agency Performance Rating: Not Applicable

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The M&E contained no institutional monitoring of the implementation of the laws and regulations during the project period (with the exception of the monitoring of performance contracts by the MOH). Instead, the DPL drew on M&E arrangements that were set up under the Social Services Delivery Project (P055383) and the Health Systems Modernization Project (P082814). DPL outcome indicators had been defined, but would provide relatively limited information about the effects of the legislation/regulations. While baselines were set where appropriate, monitoring on a continuous basis does not appear to have been feasible, except to the extent monitoring was provided through ongoing investment projects. This is reflected in the fact that only two outcomes were measured, one of which (the one in health) was ambiguous, and the other (disability) indicated a moderate decrease in the budget for disability, but gave no indication of actual spending.

b. M&E Implementation:

Monitoring of the DPL was based on the two supervision missions in the eight months from effectiveness to closing of the DPL operation, and supplemented by ongoing interaction with the implementing agencies on the programmatic support in the two policy areas addressed through the DPL, as well as the two ongoing investment operations. Monitoring (or evaluation) of the implementation of a set of laws is important, but their impact can hardly be assessed in a meaningful way over a period of only one year. Instead, monitoring of the reforms supported in the DPL, would be taking place over a multi-year period by the Government and in ongoing investment loans, in the context of medium-term reform in social protection and health.

a. M&E Utilization:

Monitoring of prior actions under the DPL fed into part of the monitoring of overall reforms in the two sectors. Outside that context, monitoring would appear less meaningful. Still, findings related to key policy actions did influence the design of the Social Assistance Modernization Project that was approved on April 3, 2012.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
No safeguard policies were triggered.

b. Fiduciary Compliance:
There were no fiduciary concerns.

c. Unintended Impacts (positive or negative):
None reported

d. Other:
None reported



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Unsatisfactory
While the project was responsive to country conditions and Bank strategy, its design did not correspond well to the PDO. The choice of lending instrument and implementation time frame were inadequate for achieving the objectives. The results framework would offer little guidance on what improvements in outcomes could be expected, and in particular how the outcome indicators chosen would meet the PDO 
Risk to Development Outcome:
Moderate
Negligible to Low
Both the ICR and the IEG review agree that there was moderate macro risk at project closing in light of the economic situation in the region. This risk has since diminished.  
Bank Performance:
Satisfactory
Moderately Unsatisfactory
The relevance of project indicators to the development objectives was tenuous. The choice of lending instrument appears inappropriate in relation to both the objectives and the country's situation. 
Borrower Performance:
Satisfactory
Moderately Unsatisfactory
The DPL experienced slower than expected implementation and an absence of information to facilitate a systematic monitoring of commitments under the DPL, which in turn contributed to the fact that key targets remained unmet. 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

IEG draws the following two lessons from the experience of preparing and implementing this operation:
    • Single-tranche DPLs are likely to be appropriate instruments for supporting medium term sector reform only when such reforms require temporary budget support or generate significant fiscal costs.
    • It is unlikely that prior actions supported by a single-tranche DPL would have measurable outcomes over short periods of time. The operation under review is a case in point.

14. Assessment Recommended?

Yes
Why?
A joint Project Performance Assessment Report (PPAR) for this operation and the Health System Modernization Project (P082814), which closed on June 30 2012, would yield useful lessons on supporting social sector reform in Albania and elsewhere. The PPAR could also be used to verify the ratings in the light of developments since closure of the two operations.

15. Comments on Quality of ICR:


The ICR provides reasonable evidence for its ratings, with the exception of the macro-fiscal discussion, which does not convincingly support a DPL approach; and the school bonus conditionality, the justification for which is never assessed. The document provides an appropriate balance between narrative and analysis. It is consistent with the OPCS guidelines, and is concise.


a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6DPL-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions