Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Health System Modernization Project


  
1. Project Data:   
ICR Review Date Posted:
11/05/2013   
Country:
Albania
PROJ ID:
P082814
Appraisal
Actual
Project Name:
Health System Modernization Project
Project Costs(US $M)
 19.15  19.00
L/C Number:
C4154
Loan/Credit (US $M)
 15.40  14.50
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
 1.61  1.61
Cofinanciers:
Japan
Board Approval Date
  03/14/2006
 
 
Closing Date
09/30/2010 06/30/2012
Sector(s):
Health (66%), Compulsory health finance (19%), Central government administration (14%), Sub-national government administration (1%)
Theme(s):
Health system performance (67% - P) Public expenditure financial management and procurement (33% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Hjalte S. A. Sederlof
Robert Mark Lacey Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the project Appraisal Document (PAD), the Project Development Objective was to:
      1. improve both physical and financial access to and the actual use of high quality primary health care services, with an emphasis on those in poor and under-serviced areas as well as to diminish the unnecessary use of secondary and tertiary care facilities,
      2. increase the effectiveness of the MOH [Ministry of Health] and HII [Health Insurance Institute] in formulating and implementing reforms in provider payments and health system performance, and
      3. improve governance and management in the hospital sector.
    According to the Development Credit Agreement (DCA), the objective was to:
      1. improve the physical and financial access to, and use of, high quality primary health care services;
      2. improve the Recipient’s capacity to formulate and implement health policies and reforms in the health sector; and
      3. improve hospital governance and management.

    The statement of PDOs in the two documents are consistent. The ICRR will assess the project on the basis of the PAD as being the more specific of the two PDO statements.

    There were five key outcome targets:
      1. At least 70 percent of the population is enrolled with a primary health care provider and use him/her as their first source of health care;
      2. Reduced percentage of households who do not seek necessary health care because they cannot afford it and reduced share of household expenditure for primary care services;
      3. Increased patient satisfaction with Primary Health Care (PHC) treatment and improved outcomes for defined health conditions;
      4. HII is able to live within its overall budget, without additional allocations from the Ministry of Finance (MOF);
      5. Based on agreed key performance indicators, hospitals using new governance approaches perform better than those who do not.
    At mid-term review, two key targets – numbers 4 and 5 – were dropped, and one new target was introduced, an increase in the number of primary care visits per capita per year.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

Component A - Strengthening Sector Stewardship, Financing and Purchasing (estimated cost at appraisal - US$7.7 million; actual cost – US$4.7 million). This component would help the Health Insurance Institute (HII) to develop its capacity as the sole purchaser of health services, and would support capacity building to strengthen the stewardship roles of the Ministry of Health (MOH), the Institute of Public Health (IPH) and the Health Insurance Institute (HII). Activities would include (i) capacity building in the HII and its local branches; (ii) strengthening the policy formulation and performance monitoring functions within the MOH and the IPH; (iii) development of health information systems to support payment and management reforms; (iv) development and implementation of a system to monitor provider performance (both clinical and financial); (v) establishing a licensing/re-licensing scheme for physicians and health facilities, and an accreditation program for hospitals; (vi) development of health technology assessment capacity; and, (vii) building up MOH capacity in financial management, procurement and project coordination.

Component B - Improving PHC Service Delivery (cost at appraisal US$10.1 million; actual cost US$12.5 million). This component would support institutional reforms and limited investments aimed at improving quality of care among health care providers and in health facilities. The program would (i) facilitate registration of the population with the HII and enrollment with a primary care physician, together with related public information campaigns; (ii) build practice management capacity at the primary care level; (iii) develop and introduce clinical guidelines with an initial focus on primary care and the primary-secondary care interface and the most pressing health issues (e.g., child health, ante-natal care, respiratory infections); (iv) establish a continuing medical education system (CME) and link this to the re-licensing scheme; (v) retrain existing general practitioners and pediatricians in evidence based treatment of common conditions and rational drugs use, based on the clinical guidelines; (vi) provide basic equipment to physicians who complete the retraining program; and (vii) establish a grant facility to fund proposals from primary care providers in support of quality of care and continuum of care improvement initiatives.

Component C - Strengthening Hospital Governance and Management (cost at appraisal - US$1.3 million; actual cost 1.8 million). This component would provide initial steps to improve hospital operations and direction by focusing on (i) the development and introduction of accounting and internal control structures for hospital care providers and training in hospital management, (ii) developing the regulatory framework, including by-laws and regulations to support the move of MOH hospitals to the status of autonomous public entities; and (iii) piloting reforms of hospital management and governance structures in selected hospitals.

There were three level 1 restructurings during implementation:

The first restructuring in April 2010 (3 1/2 years after effectiveness) affected components B and C. In component B, a new national communications strategy and campaign was introduced; production of health identity cards for enrollment in primary care was financed; and the purchase of hospital equipment was included to strengthen the primary-secondary care interface. In the absence of strong Government commitment, proposed quality improvement grants were dropped from the component. In component C, privatization of all non-medical activities in regional hospitals was introduced.

The second restructuring in September 2011 included the dropping of software procurement and the registration of the population with the HII, as the design of the health insurance information system suffered from continuous delays (Components A and B, respectively). At the same time, pilot testing of the country’s hospital rationalization strategy was introduced as a new activity under Component C.

The third restructuring in February 2012 involved a second extension of the project’s Closing Date to June 30, 2012 to ensure successful completion of two big procurement packages on medical equipment.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project cost and financing: Project cost at appraisal was estimated at US$19.1 million. The actual project cost was US$19 million. Actual IDA financing was US$14.5 million; US$0.9 million was cancelled. The Borrower contributed US$2.89 million compared to an appraisal estimate of US$2.1 million. A Japanese PHRD Grant of US$1.61 million also contributed to the financing of the project.

Borrower contribution: The Borrower contributed US$2.89 million.

Dates: The project was approved on March 14, 2006 and became effective on September 7, 2006. The closing date was extended twice. The first extension of 18 months was granted as part of the first (May 2010) restructuring in order to complete: (i) the procurement of medical equipment for the regional hospitals; and (ii) the health insurance IT system, including health identification cards for ages 0-16 years old. The second extension of three months was granted in February 2012 to allow procurement of medical equipment to be completed. The project closed on June 30, 201 -- 21 months later than the original closing date of September 30, 2010.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Relevance of objectives is rated high.
The project objectives are relevant to the country’s development priorities in health and to Bank country and sector assistance strategies. They directly support Albania’s long-term health sector reform strategy aimed at achieving equitable access to appropriate care in an environment of effective sector management. This is highlighted in Albania’s long-term National Strategy for Social and Economic Development. And the 2011-2014 CAS (Report No. 54188-AL) includes as one of its three strategic objectives to “[broaden and sustain] Albania’s social gains by improving access to better quality education and health services, and increasing the effectiveness of its social protection systems.” This is reflected in the objectives of the project under review as well as the programming of a Health Sector Project planned for FY2014, which will pick up on initiatives introduced under the project under review.

b. Relevance of Design:
Relevance of design is rated substantial.
The design was relevant. It included a menu of activities that were consistent with project objectives. It drew on lessons learned from an earlier health project and from similar projects in neighboring countries. It included both primary care network expansion and rehabilitation, and challenging institutional reforms. Initially, institutional reforms that were being introduced did not appear to be well calibrated to absorptive capacity in the implementing agencies. This imbalance was to be addressed by external technical assistance that would gradually be replaced by local experts; and by local working groups to undertake coordination and technical management of the project. Overall project management was to be located in the MOH. While project design would not altogether withstand unanticipated shocks during implementation – repeated senior personnel changes in the MOH, including six Ministers, would lead to subsequent adjustments in project contents (the three project restructurings); and MOH would not turn out to have sufficient capacity to manage the project – key institutional reform activities were maintained. The results framework was inadequate: it included targets that were not achievable, and ones that were difficult to monitor correctly.


4. Achievement of Objectives (Efficacy) :


Objective 1: to improve both physical and financial access to and the actual use of high quality primary health care services, with an emphasis on those in poor and underserved areas as well as to diminish the unnecessary use of secondary and tertiary care facilities.

Efficacy is rated substantial. Physical and financial access to primary care showed measurable improvement, as did indicators of use and satisfaction, although concerns remained about the quality of care at health centers. The use of secondary and tertiary facilities declined (this may have been due to the introduction of performance-based payments and new clinical guidelines, as well as resulting from any outputs under this objective). There is no evidence on the effect on poor households.

Outputs

    1. All regional hospitals were supplied with essential medical equipment;
    2. 2,200 primary health care facilities constructed, renovated and/or equipped;
    3. Training provided to 630 primary health care providers in family medicine, and to general practitioners in the implementation of new clinical guidelines;
    4. Quality training for primary care workers was provided in three regions, compared with a target of six regions
Outcomes
    1. The number of visits per capita per year to primary care centers increased from 1.6 visits to 2.0 visits, meeting the target.
    2. Based on the results of a household survey, patient satisfaction with treatment in primary care centers rose from 88.4 percent to 98 percent, However, there is no proper data to assess if outcomes for defined health conditions have improved as a result of project activities.
    3. The share of households not seeking necessary care due to financial factors fell by 47 percent to 19 percent compared to a target reduction of 20 percent.
    4. The share of the population enrolled with a primary care provider rose from zero to 70 percent over the project period, meeting the target.
    5. The number of pregnant women receiving antenatal care during a visit to a health provider increased from 95 percent to 97 percent, and the share of immunized children rose from 94 percent to 95 percent.
Objective 2: to increase the effectiveness of the MOH and the HII in formulating and implementing reforms in provider payments and health system performance

Efficacy is rated substantial.

In the absence of relevant indicators, there currently is no available evidence of increased effectiveness. However, the outputs (and intermediate outcomes) related to this objective can reasonably be expected to increase the effectiveness of the MOH and HII in formulating and implementing reforms in provider payments and health systems performance. The measures for institutional strengthening of the MOH are all aimed at improving health system performance. The strengthening of the HII as sole purchaser of health services establishes a provider payment system that, combined with a unified benefit package, provides a predictable and monitorable environment for the provision of primary care.

The Health Insurance Institute:

Outputs

1. Management capacity was developed in HII to act as sole purchaser of health services;

2. A performance-based payment system and a unified benefit package were developed and applied to primary health care providers contracted by HII.

3. Hardware was procured for the development of an information system on participants in the HII scheme.

Outcomes

1. The establishment of HII as sole purchaser of health services has reduced fragmentation in financing and created the necessary environment for performance-based incentives for providers;

2. Performance contracts have been established with 420 health centers, 2,000 family practitioners, and 6,600 nurses;

3. The information system is being used for reporting/assessing primary care activities based on performance indicators.

The Ministry of Health:

Outputs

1. A long-term health sector strategy was developed;

2. A monitoring and evaluation capacity was set up with responsibility for strategic planning, and monitoring and evaluation of the health system;

3. A licensing and accreditation system has been introduced for all providers;

4. Evidence-based clinical guidelines have been issued

Outcomes

1. MOH has improved capacity to monitor, implement and adjust the sector health strategy over time;

2. The introduction of clinical guidelines has resulted in the establishment of protocols in primary services;

3. All public health facilities have been accredited.

Objective 3: to improve governance and management in the hospital sector

Efficacy is rated modest.

A total of 230 hospital managers and other staff in tertiary and district hospitals were trained in general and financial management. A coherent national rationalization plan for hospitals was developed, including (i) strengthening diagnostic capacity in a limited number of regional hospitals and (ii) integrating others with primary care facilities; the plan, together with reforms in hospitals management, was piloted in two regions. A plan for outsourcing non-medical services in regional services was developed. And a framework was developed for hospital autonomy.

Again, there are no indicators of efficacy.


5. Efficiency:


Efficiency is rated substantial.

The PAD (Annex 9) contained a cost-benefit analysis. The ICR also undertook a cost-benefit analysis. Due to difficulties in estimating Components A and C, both analyses focused on Component B – Improving PHC Service Delivery – covering about 50 percent of total project costs at the appraisal stage and 59 percent at project closing. Benefits were to be derived from better PHC services and a reduction in unnecessary hospital admissions and specialist services. The economic rate of return at appraisal was estimated at 12.9 percent and at 18 percent by the ICR. The difference is mainly generated by reductions in hospital admissions and referrals to specialists (minus 2.4 percent at appraisal vs. minus 5 percent at completion).

Project implementation suffered delays, requiring two restructurings including modification of some components and revisions to the Results Framework, and leading to an extension of the Closing Date by 21 months. All three components were modified, leading to a reduction in the base cost of the first component from US$5.02 million to US$4.41 million; an increase in the second component from US$9.07 million to US$10.06 million; and an increase in the third component from US$1.31 million to US$1.46 million. Total baseline costs fell from US$16.37 million to US$15.93 million.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
12.9%
52%
ICR estimate:
Yes
18%
66%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Relevance of objectives is rated high, as it accorded with both government policy and Bank country assistance strategy. The relevance of design was related substantial, reflecting an ambitious, but balanced, design, albeit with a weak results framework. Efficacy was rated substantial for two of the three objectives, and modest for the third objective, reflecting reasonable performance under the first two objectives, and the absence of measurable outcomes for the third objective. Efficiency was rated substantial, based on a rate of return that exceeded expectations at the beginning of the project, and a baseline cost lower than that estimated at appraisal.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:


At the time of the ICR, there appeared to be sufficient support for reform measures introduced under the project, but the ICR judged capacity in the MOH to operationalize essential components of the second development objective -- licensing and accreditation, technology assessment, clinical guidelines and hospital rationalization -- still to be insufficient without donor support. While donor support is expected to be forthcoming, this was not yet the case at project completion.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:

The project was strategically relevant: the main features of project design were consistent with government priorities and health strategies. The project drew on prior experience in Albania and neighboring countries. Its technical design was ambitious, but this was addressed through collaboration with national working groups and by focused technical assistance during implementation. The project team included the necessary skills mix, and it was given sufficient resources in time and money for thorough preparation (as reported in the ICR). Implementation arrangements would turn out to be less successful. The working groups that were to provide continuity in coordination and technical management of the project were not introduced; and the MOH, who was to be in charge of overall management, would turn out not to have sufficient capacity. The results framework would turn out to be insufficient for rigorous monitoring and evaluation, and would have to be revised during implementation. Risks related to government commitment and political stability appear not to have been sufficiently assessed, leading to problems during implementation, reflected in project restructurings and delayed disbursements.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

Budget and staff resources were sufficient to allow thorough supervision (according to the task team), and missions were able to maintain close consultations with the client and with donors. They were able to detect problems as they arose, and in most instances work flexibly with the government and implementing agencies to find solutions/alternatives while maintaining the project’s focus on the PDOs. The effectiveness of supervision was, however, reduced by numerous personnel changes in the MOH, including at Minister level. These changes were accompanied by changes to project components and the absence (until the mid-term review) of local working groups that were supposed to be in place to draw the Borrower’s attention to, and address, technical issues that arose during implementation.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government maintained its formal commitments to the Bank as specified in the Credit Agreement. Its actions did, however, generate two problems that adversely affected project implementation: (i) technical working groups that were to form the link between national stakeholders and Bank teams in addressing technical issues as they arose, were established only after mid-term review, weakening feedback and corrective mechanisms; and (ii) frequent changes in leadership in the MOH caused disruptions in operations, especially as leadership changes generated subsequent changes among senior staff dealing with the project.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:

Implementing agencies were MOH and HII. To compensate for limited capacities in the two agencies, a Project Implementation Unit was established in the MOH. MOH worked well with the Bank teams in aligning project activities with needs as they were identified, but implementing agency performance suffered from a number of problems: (i) changes in leadership in the MOH, as the Minister was changed six times and subsequently all senior staff engaged in the project were replaced, causing disruptions in project implementation; (ii) weak project management capacity in the MOH (which was balanced by the introduction of a separate project implementation unit; (iii) lack of coordination between the MOH and the HII, resulting in an information technology component that was only partially completed; (iv) cancellation of the quality improvement grants; and (v) procurement problems (see Section 11 below).

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

A comprehensive monitoring and evaluation strategy was developed, with baseline data to be obtained through a household survey of the key sector issues, supplemented by a subsequent Living Standards Measurement Survey (LSMS) to provide key inputs on patient satisfaction, out-of pocket payments and delays in care-seeking behavior. As part of the project, capacity was to be built in MOH and HII to facilitate the mainstreaming of monitoring and evaluation, both for the project interventions and for the health system generally. Annual performance reviews were to monitor progress with respect to project implementation and impact. The same approach was to be developed to promote annual sector performance reviews as a base for determining policy achievements in relation to the overall health sector strategy.

b. M&E Implementation:

The results framework, indicators and targets were revised during implementation, in part because they were too ambitious or difficult to monitor, and in part because project components were revised. The MOH collected data to track progress towards outcomes, but experienced difficulties due to delays in carrying out the household survey and the LSMS. As a result, a number of original indicators, or ones introduced during project revisions, could not be used.

a. M&E Utilization:

Project data was used to track implementation progress and to inform policy makers about health strategy. The gradual institutionalization of M&E into the MOH and HII allowed these agencies to monitor the effectiveness and quality of services at different levels of the system. This is currently being used for further health sector planning and development.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
No safeguard policies were triggered by this Category C project.

b. Fiduciary Compliance:

Financial management. The Project’s financial management arrangements were satisfactory throughout the implementation of the Project – necessary internal controls were in place, as well as adequate disbursement, staffing, reporting and auditing arrangements. Audits of the Project’s financial statements were completed on time and results did not show any major irregularities.

Procurement. The implementing agency (MOH), even with the assistance of a Project implementation unit, had limited procurement capacity and faced problems in preparing bidding documents for medical equipment. A number of complaints were registered, and in one instance, the Bank had to cancel one lot of medical equipment in the amount of US$ 1.1 million.

c. Unintended Impacts (positive or negative):
None

d. Other:
None



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Satisfactory
The ICR recognized successful project implementation, but also stressed the absence of indicators that would allow effective verification of targets, especially those for patient satisfaction and health status. The IEG review recognizes similar shortcomings, but places more importance on the project's substantive achievements in building key pillars of an effective health system: the introduction of a public single payer; performance management; and measurably improved physical and financial access to services. These are seen as an essential springboard for improved health status over time. 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Satisfactory
Moderately Satisfactory
There were weaknesses in stakeholder analysis and results framework that made implementation difficult; and were corrected during implementation 
Borrower Performance:
Moderately Satisfactory
Moderately Unsatisfactory
There were significant shortcomings in both Government and Implementing Agency performance, including absence of commitment and engagement in MOH, frequent changes in leadership, and weak management and technical capacity 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The following lessons are taken from the ICR with some adaptation of language:
    • Complex reforms require a well-sequenced roadmap. Projects that are comprehensive quickly become unmanageable, since they try to do too many things over too short a period of time. Particular attention needs to be paid to developing implementable plans, or roadmaps, that recognize constraints and challenges, and absorptive capacity, and allow for appropriate pacing.
    • The capacity of the Government to absorb technical assistance and implement complex institutional changes needs to be carefully evaluated to ensure knowledge transfer and a sustainable impact of the activities started under a project.
    • The development and integration of health information systems in the health sector is complex. Dedicated teams that work in close collaboration with information users is essential; and should be set up as the project is being prepared. Technical assistance should be timely and integrated into such teams.

14. Assessment Recommended?

Yes
Why?
A joint Project Performance Assessment Report (PPAR) for this operation and the Social Sector Reform Development Policy Loan (P116937), which closed on April 30 2012, would yield useful lessons on supporting social sector reform in Albania and elsewhere. The PPAR could also be used to verify the ratings in the light of developments since closure of the two operations.

15. Comments on Quality of ICR:

The ICR provided sufficient, balanced, evidence to make an assessments of the project’s outcomes. It provided strong analytical sections on the Key Factors Affecting Implementation; and it undertook an economic analysis. It was consistent with the OPCS guidelines. It could have been more concise.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions