Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Health Sector Development Project


  
1. Project Data:   
ICR Review Date Posted:
05/13/2013   
Country:
Georgia
PROJ ID:
P040555
Appraisal
Actual
Project Name:
Health Sector Development Project
Project Costs(US $M)
 24.8  26.9
L/C Number:
C3702
Loan/Credit (US $M)
 20.3  24.3
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
 0  0
Cofinanciers:
Board Approval Date
  08/01/2002
 
 
Closing Date
12/31/2007 06/30/2012
Sector(s):
Health (80%), Central government administration (15%), Compulsory health finance (5%)
Theme(s):
Rural services and infrastructure (29% - P) Health system performance (29% - P) Population and reproductive health (14% - S) Child health (14% - S) Law reform (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Judith Hahn Gaubatz
Judyth L. Twigg Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Project Appraisal Document (PAD, page 2) and the Development Credit Agreement (DCA, page 14), the objective of the project was:
      • To improve the coverage and utilization of quality primary health care based on the model of family medicine/general practice, with an emphasis on reaching the poor and disadvantaged.

    The project objective was revised in May 2009, according to the Project Paper (page 5), as follows:
      • To improve coverage, utilization, and quality of health care services in the territory of the Recipient; and
      • To strengthen the government's stewardship functions in the health sector.

    The project name was also changed to the "Health Sector Development Project" to reflect the broader scope of the project.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 05/20/2009

c. Components:

Original Project:
1. Primary Health Care Service Delivery (Appraisal: US$16.2 million; Actual: n/a): This component was to support the development of primary health care (PHC) services in rural and urban areas through rehabilitation and equipping of clinics. It was to be implemented in two phases: Phase I would focus on upgrading PHC clinics in up to 74 rural and high mountain areas of the Imereti region; Phase II would expand support to PHC clinics in urban areas and other rural areas, provided that adequate progress was made in Phase I. Activities were to include: rehabilitation of PHC clinics and referral laboratories; provision of diagnostic and medical equipment; a pilot program to test the referral network for maternal and child health services through the Kutaisi Maternal and Child Health Center; and information and education campaigns to increase public awareness of family medicine and PHC services.

2. Institutional Development (Appraisal: US$7.1 million; Actual: n/a ): This component was to support capacity building in medical training and health care management; as well as development of a policy framework and regulatory environment for primary health care (PHC). Activities were to include: rehabilitation of five Regional Family Medicine Training Centers; equipping of facilities for family medicine faculty and a residency program; stipends for doctors and nurses participating in family medicine re-training programs; support to the Ministry of Labor, Health and Social Affairs (MOHLSA) for planning and regulating PHC services, including developing a master plan for PHC clinics; strengthening the health management information system; and establishing of a Health Policy Unit, which would address health care financing issues.

3. Project Management Support (Appraisal: US$1.5 million; Actual: n/a): This component was to support the operations of a project management unit, called the Health and Social Project Implementation Center. This unit was to be established as a legally independent entity within the Ministry of Labor, Health and Social Affairs, with clearly defined project management functions (i.e. procurement, M&E).

Restructured Project:
The objectives and activities of the project components remained essentially the same in the restructured project, although allocations were revised and some of the activities were shifted among the components. (Note: The ICR reports the actual costs per component for the Credit only, not for the total project including the counterpart amounts.)

1. Strengthening PHC System (Revised: US$17.8 million; Actual: US$11.2 million): Information and education campaigns were to be shifted to Component 2, as part of efforts to increase public awareness about the government's overall health sector reform program. The pilot program to test the maternal and child health referral network was to be shifted to another donor.

2. Support for Health Sector Reform (Revised: US$4.1 million; Actual: US$11.2 million): Support for capacity building in the Ministry of Labor, Health and Social Affairs was to be increased, including a stronger focus on monitoring and evaluating sector performance, and developing national health accounts. Training of medical personnel on PHC services was to be shifted to Component 1.

3. Project Management (Revised: US$1.6 million; Actual: US$1.9 million): No significant changes.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Cost:

    • The original project cost was appraised at US$24.8 million. Under the project restructuring, the appraised project cost revised to US$28.0 million, due to exchange rate fluctuations. The actual project cost was US$26.9 million.
    • Costs for Component 1 (PHC service delivery) were significantly lower than appraised due to a shift in government policy to privatize PHC clinics. Costs for Component 2 (Institutional development) increased significantly due to the strengthened project focus on capacity building in the Ministry of Labor, Health and Social Affairs.

Financing:
    • The project was originally financed by a Credit of US$20.34 million. Under the project restructuring, the appraised Credit amount was revised to US$23.5 million, due to exchange rate fluctuations. The actual Credit disbursed was US$24.28 million.

Borrower Contribution:
    • The planned borrower contribution (for both the original and restructured project) was US$4.5 million. Counterpart funds were provided in full and in a timely manner.

Dates:
    • In March 2009, the project was formally restructured in its title, objectives, results framework, and components, with the project closing date extended to December 2010.
    • The original project closing date of December 2007 was extended on four occasions to December 2009, December 2010, December 2011, and June 2012. The extensions were granted to allow completion of activities.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Original: High
Revised: High
The country's health system was largely based on the Soviet system, which consisted of a large number of hospital facilities and specialized health personnel, with low emphasis on preventive and primary health care services. There were sharp declines in public expenditures on health (from 13.2% of total public expenditures in 1991 to 0.54% in 1994) and inefficiencies in the existing health system (as reflected by the above-mentioned emphasis on hospital-based care, as well as the high proportion of private out-of-pocket payments (73.0% in 2000)). Therefore, the original project objectives focused on re-orienting the health system towards primary health care and addressing inefficiencies in public expenditures on health. The Bank's current Country Partnership Strategy (FY10-13) includes improved health coverage and improved public resource management (including for key social programs targeted to the poor) as key results areas. The country's medium-term development strategy, laid out in 2008, specifically identifies expanded coverage of health insurance, particularly for the poor and the vulnerable, as a key priority.

The additional revised objective to strengthen the government's stewardship of the health sector was also highly relevant, given the aforementioned inefficiencies in public health expenditures.

b. Relevance of Design:

Original: Substantial.
Revised: Substantial.
The original project design included upgrading primary health care (PHC) clinics, as well as increasing demand for PHC services through communication campaigns. These outputs were likely to contribute to the intended outcomes on primary health care. Implementation of activities in the rural (remote) regions ensured targeting of the poorest and disadvantaged households. Although unforeseeable, a significant shift in government policy in 2008 (from the direct delivery of health services to policy making, regulation, and execution of public health functions) led to privatization of health facilities and therefore rendered direct project support to PHC clinics less relevant. The project design did not adequately address the financial incentive structures that would lead to increased demand for the upgraded facilities.

The revised project design continued support to the remaining key project activities: capacity building in family medicine training and development of a policy and regulatory framework to support primary health care. The revised design also increased focus on the capacity of the Ministry of Labor, Health and Social Affairs to monitor, analyze, and evaluate the performance of the health sector.


4. Achievement of Objectives (Efficacy) :

A number of project activities covered the entire country (medical training and sector management), while the infrastructure improvements (primary health care clinics) took place in rural areas. According to the project team, the infrastructure improvements were implemented in the four poorest rural regions in the country (Imereti, Shida Kartli, Adhara, Racha). The estimated population of the four regions is 1.46 million, out of 4.2 million total population.

Other donors, most notably the UK Department for International Development (DFID) and the European Union, were also actively engaged in primary health care development in the country; however, DFID implemented activities in urban areas and the European Union implemented activities in rural areas not covered by the Bank project. Observed results here can therefore reasonably be attributed to project-financed interventions.

To improve coverage of primary health care (Original and Revised)
Substantial, due to evidence of improved coverage of primary health care (PHC) clinics and family medicine staffing.

Outputs

    • Construction and/or rehabilitation of 103 rural PHC clinics in Imereti, Adjara, and Shida Kartli regions by the end of 2007. The original target was to construct/rehabilitate 180 clinics. However, by March 2008, the government had approved measures that would privatize rural health practices.
    • Provision of medical supply packages to 98 rural PHC clinics in Imereti, Guria, Adjara, Shida Kartli and Racha-Lechkumi regions.
    • Construction of a 25-bed hospital in Ambrolauri.
    • Refurbishment of a diagnostic laboratory in Kobuleti.
    • Development of an information, education, and communication strategy promoting primary health care. Most activities were carried out in the period May to December 2007.

However, the pilot activity to develop a maternal and child health referral center was only partially implemented. The Kutaisi Perinatal Center was partially rehabilitated and equipped; however, training was shifted to another development partner.

Outcomes
    • In 2009, the proportion of the rural population that had access to a PHC clinic within 30 minutes of walking/other transportation was 71%. No baseline is provided.
    • 41% of the total population had access to family medicine doctors (of which 86% had received re-training) and 49% by family medicine nurses (of which 100% had received re-training). The target was 50%.
    • The DPT3 immunization coverage rate for infants (nationwide) increased from 78% in 2004 to 92% in 2010, achieving the target of 90%.


To improve utilization of primary health care (Original and Revised)
Modest, due to limited evidence of improvements in utilization rates.

Outputs
    • See outputs reported above on upgrading of PHC clinics and information and education campaigns.

Outcomes
    • The number of out-patient visits per capita per year in project areas was 1.8 in 2010. The target was 2.3 for the general population and 2.6 for the poor. There was no baseline figure provided specifically for the project areas.
    • The proportion of pregnant women who have had at least four prenatal visits (nationwide) increased from 83% in 2007 to 91.3% in 2010.

The ICR (page 13) suggests several reasons for the modest improvements in utilization of PHC services. The Medical Insurance Program for the Poor did not substantially cover services at PHC clinics, and it did not provide for free or low-cost medicines. Financial access barriers were also significantly lowered for hospital services, where quality of services is perceived to be better and medicines are covered by insurance, and therefore patients may have been more likely to choose hospital services over primary care.
    To improve quality of primary health care (Original and Revised)
    Substantial. A number of project activities regarding training and facilities upgrading were implemented, and there was some evidence of improvements in quality.

    Outputs
      • Rehabilitation and/or equipping of five regional family medicine training centers.
      • Support to the family medicine training program, including equipping of faculty facilities, developing a residency program, and conducting workshops/study tours.
      • Development of a family medicine nursing curriculum, which was used in the five regional training centers.
      • Training of 916 doctors and 1,073 nurses in family medicine. The total number of family medicine doctors needed in the country for adequate coverage was 2,200, of which half were targeted to be trained during the project period.
      • Training of 404 primary health care doctors in health care management.
      • Development of 20 nationally or internationally approved treatment guidelines. 550 rural doctors received training in the new clinical guidelines, achieving the target of 550.
      • See also outputs reported above on rehabilitation and/or equipping of PHC clinics.

    Outcomes
      • According to a 2010 health utilization survey, 77.4% of the population reported that the reasons for treatment were completely explained; 90.3% reported that adequate time (more than 12 minutes) was spent with primary medical professionals; and 95.4% reported clean or very clean facilities. However, these figures represent nationwide data and not the project area specifically, and there are no comparable baselines.
      • 52% of tuberculosis patients were being treated according to the internationally approved DOTS strategy, achieving the target of 40%.
      • See also outcomes reported above on increased coverage of immunizations and antenatal care.


    To strengthen the government's stewardship function in the health sector (Revised)
    Substantial, due to the delivery of numerous project outputs and subsequent implementation of key health policies.

    Outputs
      • Development of the PHC master plan, which included a management and organizational framework for PHC, as well as health work force rationalization and health facilities (PHC clinics and laboratories) rationalization plans.
      • Development of a Health Sector Strategy White Paper for 2011-2015, the first since 1999.
      • Development of a National Pharmaceutical Policy, amendment of legislation to allow parallel imports of medicines, and preparation of an Essential Drug List (based on World Health Organization guidelines).
      • Establishment of a Health Policy Unit, which was responsible for health care financing policy. The Unit developed the capacity to prepare annual National Health Accounts, which the ICR (page 17) reports have been used to inform health financing policy and allocations of government funds.
      • Conducting of a Health Sector Performance Assessment, with findings disseminated.
      • Conducting of two rounds of Health Utilization and Expenditure Surveys (2007 and 2010).
      • Evaluations of the Medical Insurance Program for the Poor (MIP) in 2008 and 2010.
      • Implementation of information and education campaigns to increase public awareness of the government's health reform agenda, specifically programs for the poor, privatization of health facilities, and contracting with private insurance.
      • Implementation of policy to increase remuneration for family medicine personnel.
      • The health management information system was implemented as a pilot, but is not yet fully operational on a nationwide scale. A set of regulations on the standardization of procedures and codes for the information system have been submitted to Parliament for approval.

    Outcomes
      • 38.4% of public health expenditures were earmarked for programs for the poor, according to the National Health Accounts for 2011. This surpassed the target of 30.0%.
      • 93.2% of the population was aware of the Medical Insurance Program for the Poor (MIP), an indication of public awareness of health care reforms.
      • The amended pharmaceutical legislation led to a decrease in prices of some original brand medicines.
      • The MIP evaluation results led to a revision of the insurance package, adding pharmaceutical benefits. The Essential Drug List was used to inform the contents of the pharmaceutical benefits.
      • The NHA was used for preparing the medium-term expenditure framework for the health care budget.

5. Efficiency:

Modest. The project design featured some elements that were likely to contribute to efficiency in the use of project resources. These included the updating of treatment protocols and the strong focus on primary health care delivery, which was significantly less costly than hospital-based or specialized clinic care. However, there were only modest improvements in utilization rates for primary health care, in part due to the inadequate focus on changing financial incentives to utilize primary care services. In addition, numerous policy and leadership changes by the government led to significant implementation delays, with the project period extended numerous times for a total of 4.5 years.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Original: Moderately Satisfactory.
The relevance of the project objectives was rated High and the relevance of the project design was rated Substantial. The achievement of the objectives to increase coverage and to increase quality of primary health care were rated Substantial. However, there were shortcomings in the achievement of the objective to improve utilization of health care services (due to insufficient evidence of outcomes). Efficiency was rated Modest.

Revised: Moderately Satisfactory.
The relevance of the project objectives was rated High and the relevance of the project design was rated Substantial (due to improved alignment with government health policy). Although there remained shortcomings in the achievement of the objectives to improve utilization and to improve quality of primary health care services, achievement of the additional objective to strengthen the government's stewardship function for the health sector was rated Substantial. Efficiency remained Modest.

Overall: Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

Although external donor funding (including Bank funding) is decreasing for the health sector, the government intends to continue in the direction of the current health sector reforms, which are reflected in the new National Health Strategy for 2011-2015. Medical personnel training activities are being absorbed by the medical institutes, while capacity building efforts in the Ministry of Labor, Health and Social Affairs are also likely to be sustained through continued use of project outputs (i.e. national health accounts, surveys, policy and legal frameworks). Technical assistance from the Bank will be continued through policy dialogue and analytic work. The upgraded level of health facilities will also likely be sustained through the privatization process, which had factored in maintenance costs of PHC clinics in the lump sum amount provided to private doctors.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:

The project design drew on lessons of experience (hospital optimization, family medicine, and community health insurance schemes) and analytic work (poverty assessments, public expenditure reviews, and the government's health care strategies) in the country and the region. However, a shortcoming in the project design was the failure to adequately address financial incentives to increase use of primary health care services. The overall risk rating was Substantial, with risks related to the unstable political-economic environment, inadequate public financing, civil works cost inflation, and lack of local level participation in PHC development considered Substantial. However, the risk of changes in political leadership and resulting discontinuation of the reforms was assessed as Negligible, and yet this risk subsequently materialized with significant consequences for the original project objectives and design. The safeguard requirement to prepare and publicly disclose the environmental management plan was unmet. There were also shortcomings in the original M&E framework (see Section 10).

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

Supervision missions were conducted regularly and with the appropriate skills mix (including both fiduciary and technical specialists). The presence of a field-based task team leader helped to ensure timely implementation support. In response to the significant shift in the government's health sector reform agenda in 2006, the Bank team worked to restructure the project and sustain its engagement in the health sector. The results framework was revised accordingly, in part to reflect the changed objectives but also to refine the existing indicators for improved measurability. An assessment was conducted of all outcome and output indicators during the restructuring mission, with the revised indicators more closely aligned with project activities and expected outcomes. There was overall effective collaboration with other donor partners to ensure consistency in approaches to health sector reforms and limited overlap in activities. Fiduciary performance was satisfactory with no major financial management or procurement problems reported. However, a shortcoming was the apparent lack of oversight on safeguard compliance. Although there was a lack of documentation on implementation of safeguard measures (see Section 11), there were no indications of problems with environmental impacts.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

In 2006, the government signalled a significant shift in its health care reform agenda, which led to disruptions in project implementation and affected the relevance of some project activities. Despite the ongoing conflict between Georgia and the Russian Federation, which led to temporary suspensions of procurement and delays in project activities, the government sustained its commitment to health reform overall, specifically in supporting the legal and policy frameworks. Leadership of the Ministry of Labor, Health and Social Affairs changed eight times over the project period, often accompanied by reorganization, leading to delayed decision-making and the need to rebuild project ownership. The project restructuring process was also postponed several times due to political events. Counterpart funds were provided in a timely manner.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

The Ministry of Labor, Health and Social Affairs, despite changes in leadership and policy directions, was overall effective in managing the project and ensuring implementation of project activities. Fiduciary performance of the project management unit was satisfactory, with no major financial management or procurement problems reported. The clearly defined functions of the project management unit, as well as its independent status, enabled it to continue project operations despite periods of political instability. However, as noted previously, a shortcoming was the lack of documentation to verify that safeguard measures were implemented as required (see Section 11).

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The original M&E indicators appropriately reflected the project's intended outcomes; however, the preciseness and measurability of some of the indicators was inadequate. For example, the original indicator of "proportion of the population (with access to PHC services) completing at least three visits per capita per year" was changed to "number of outpatient visits per capita per year"; "20% increase in the proportion of infants that receive DPT3 immunization on time" was changed to "increased DPT3 immunization rate." These shortcomings were addressed during the project restructuring when the results framework was revised; the extent of the revisions indicates the inadequacy of the original design. 9 out of 17 indicators were modified, 7 were dropped, and another 17 were added. The M&E design also included evaluative activities such as surveys and impact assessments. Data were to be collected by the project management unit.

b. M&E Implementation:

The project management unit, in collaboration with the Medical Statistics Unit, was able to collect data on the numerous project indicators and provide systematic reports on project indicators. Surveys (Health Utilization and Expenditure Surveys) and assessments (Medical Insurance Plan for the Poor, Health Sector Performance) were conducted as planned. The health management information system was designed, although not fully implemented as of the project closing.

a. M&E Utilization:

The ICR (page 20) reports that, despite the systematic collection of data by the project, there was only limited use of the M&E information generated.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

The project was classified as a Category "B" project during negotiations. Although the project appraisal document required preparation of an environmental assessment (and an environmental management plan), the ICR (page 21) reports that a copy of the plan could not be located, nor was there any evidence that the plan had been discussed with stakeholders and publicly disclosed. The ICR suggests that lack of clarity in the Bank's safeguards handbook about Bank disclosure requirements may have caused confusion. The ICR also indicates that there was no record of supervision of environmental management during the project period, including any regular reports on safeguard compliance in the mission aides-memoire (the only exception being a brief reference to "a safeguards effectiveness review" conducted during the period April 6-10, 2007 in the Mid-Term Review supervision mission aide-memoire). However, the project team noted that environmental guidelines were provided to all contractors, all civil works contracts included clauses about environmental issues, and there were no indications of negative environmental impact.

b. Fiduciary Compliance:

Financial management: A financial management review was conducted during project preparation, and financial management capacity and arrangements were considered satisfactory. Staffing and financial reports were adequate throughout the project period, and no major problems were reported. Annual audits were timely and unqualified.

Procurement: The project management unit was adequately staffed with procurement specialists, and no major delays or problems were reported in procurement performance. A fiduciary review, conducted by the Bank in 2011, reported generally satisfactory procurement implementation, including up-to-date procurement planning, timely implementation of procurement activities, and general compliance with Bank procedures.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

Lessons drawn from the ICR, with adaptation by IEG:
    • The proper project management arrangements can enable a project to sustain operations in the midst of a politically unstable environment. In the case of this project, the project management unit, which had been established as an independent entity within the Ministry of Labor, Health and Social Affairs, had clearly defined project management functions and continuity of experienced staff; therefore, it was able to continue implementation despite frequent changes in government leadership and sector reform priorities.
    • In an environment of ongoing (and rapidly shifting) reforms, restructuring the project can be an avenue to sustain engagement in the sector; however, this needs to be balanced against the time and resources expended in restructuring.

14. Assessment Recommended?

Yes
Why?

    To verify outcomes and to draw lessons on sustaining engagement in the context of shifting policy agendas, as well as on the importance of addressing financial incentive structures in health reform agendas.

15. Comments on Quality of ICR:


The ICR quality is satisfactory overall. The quality of the evidence is adequate, in particular the detailed reporting of project outputs. However, there are some shortcomings in the quality of the analysis, as the numerous project outputs were not convincingly linked to the reported outcomes. For example, some of the outcome data reflect nationwide figures, while the project supported activities only in rural areas.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions