Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - First Health


  
1. Project Data:   
ICR Review Date Posted:
09/09/2005   
PROJ ID:
P009125
Appraisal
Actual
Project Name:
First Health
Project Costs(US $M)
 69.8  82.3
Country:
Uzbekistan
Loan/Credit (US $M)
 30.0  29.6
Sector, Major Sect.:
Central government administration, Sub-national government administration, Health,
Law and justice and public administration; Law and justice and public administration; Health and other social services
Cofinancing (US $M)
 0  2.69
L/C Number:
L4396      
   
Board Approval (FY)
  99
Partners involved
USAID, British Know How Fund/DFID 
Closing Date
06/30/2003 12/31/2004
         
Evaluator: Panel Reviewer: Group Manager: Group:  
Denise A. Vaillancourt
Fernando Manibog Alain A. Barbu OEDSG

2. Project Objectives and Components:

a. Objectives
The project was designed to support and inform national health reform through pilot activities in three oblasts with diverse socio-economic characteristics and a combined population of about 4 million. Lessons from pilots were to be used to replicate beneficial reforms and develop broader-based national programs to reform health care financing, organization of care and service delivery. Its specific objectives were to:

at the regional/oblast level
(a) improve the quality and cost effectiveness of primary health care, particularly in the rural areas where 70 percent of the population resides;
(b) develop a new cadre of medical personnel through short-term training programs for general practitioners and universal nurses, and in the longer-run by redefining medical education curricula;
(c) strengthen the management and financing of primary health care services, in part through incentive restructuring and decentralization in both the financing and provision of health care;
at the national level
(d) build capacity in the MoH to evaluate and disseminate the results of the pilot country-wide;
(e) guide the Government's effort to strengthen the primary health care services in the country as a whole.

b. Components (or Key Conditions in the case of Adjustment Loans):
(1) Strengthen Primary Health Care Services in Rural Areas (cost at appraisal: 53.8 million; actual cost: $71.4 million or 133 percent of estimate) through (a) construction, consolidation and rehabilitation of rural medical centers (SVPs); (b) upgrading of services (clinical, primary and preventive care, child health services, reproductive health and emergency care), including provision of drugs, medical supplies, logistical support; and (c) health promotion including communications equipment, technical assistance and training.
(2) Training of General Practitioners and Universal Nurses (cost at appraisal: $6.8 million; actual cost: $7.3 million or 107 percent of estimate) in the three oblasts including: (a) retraining of existing physicians; (b) continuing medical education of SVP physicians; (c) training of trainers of existing physicians; (d) long-term reforms for strengthening medical education; (e) development of training sites; (f) training of universal nurses; and (g) accreditation, certification and licensing. Project support was to include civil works, goods, equipment, consulting services, drugs and training.
(3) Strengthening of Financing and Management of Primary Health Care Services (cost at appraisal: $3.9 million; actual cost: $2.0 million or 51 percent of estimate): (a) pilot demonstrations in three oblasts to test rationalization of the delivery system and new financing and management models; and (b) pilot evaluation and roll-out. Envisaged project support, aimed at improving incentives, efficiency and sustainability of the Government's rural health care reform, included computer equipment, advisory services and training.
(4) Project Management (cost at appraisal: $5.3 million; actual cost: $1.6 million or 30 percent of original estimate), including support to project implementation bureaus at the central level and in the three oblasts and funds for preparation of a follow-on operation. Project financing was to cover training, consultant services, equipment, salaries and logistical support.

c. Comments on Project Cost, Financing, Borrower Contribution, and Dates
The total project cost was $82.3 million or 118 percent of the original estimate of $69.7 million. The increase in overall costs is largely attributable to a larger than planned coverage of the project, in particular a much higher number of SVPs reconstructed/rehabilitated and equipped. Loan disbursements amounted to $29.6 million or 99 percent of the loan amount. Government counterpart funding was higher than planned. The project's closing date was extended by 18 months (from June 30, 2003 to December 31, 2004) to allow for full implementation of the originally planned project activities. This was necessary because of implementation delays caused by low institutional capacity of government agencies, and some resistance by the Government to address combersome government regulations and implementation procedures, particularly in the area of procurement and delivery of goods and services.


3. Relevance of Objectives & Design:

Overall relevance of project objectives is substantial.
Objectives. Project objectives were relevant to the 1998 CAS, which laid out the challenge of removing the inefficiencies in resource utilization with particular emphasis on rural infrastructure and social services. The strategy also advocated the use of pilot and demonstration approaches to address the risks of sector reform. Health sector specific strategy sought to enhance efficiency in the use of constrained resources, improve the quality and cost effectiveness of services, and target vulnerable groups. Rationalization of health financing and health education and promotion were also highlighted. Project objectives were also relevant to Government's sector strategy, which was evolving towards an increased emphasis on primary health care, vulnerable populations (women, children, those with poor access) and a move away from the old Soviet model of expensive, inpatient medical care in favor of improved coverage, quality and cost-effectiveness of basic services.
Design. To a certain extent the design was relevant to key policies and institutional reforms that GoU was embarking upon, notably: decentralization of decision-making; incentive-based financing of care; upgrading of the medical profession; closure of excessive and redundant facilities. It was also appropriate in its limitation of the project scope, excluding support of domestic production of drugs and vaccines, requested by Government, but found to be cost-ineffective. The design showed prudence in limiting its coverage to three oblasts, widely divergent in their socio-economic indicators for a rich pilot experience. It reflected extensive consultation with a wide range of stakeholders, years of policy dialogue and analysis (including burden of disease analysis), and assessment of regional and country-specific implementation lessons. Notwithstanding these good design features, the project concept was underdevelopped in one respect. While it was conceived as a pilot operation to test sector reform implementation in three oblasts, the pilot was never well defined (operational research questions, the content and timing of each of the pilot phases: testing, evaluation, discussion/dissemination, adjustment and replication). Furthermore, the lack of baseline data and of an adequate monitoring and evaluation framework and plan undermined the project's goal of documenting and learning from the pilot experience, and applying the lessons for finetuning and generalizaing sector reform.

4. Achievement of Objectives (Efficacy) :

Serious weaknesses in M&E design and implementation (see Section 6) have made it difficult to document outcomes. Available evidence indicates that investments in oblast-level health services and management have contributed to improved service coverage, quality and efficiency. However, the learning anticipated under the pilot approach was not fully tracked or documented. The overall efficacy rating of substantial is derived from the following assessment of individual objectives.
  • Achievement of the overall objective to inform and finetune health reform implementation through a pilot approach is rated as modest. As the ICR notes (p. 16) the pilots under this project had more the character of a gradual roll-out of (PHC facility and human resources) models than really testing different service models in search of answers to specific questions. While some learning did take place through the experience of this project, it was in spite of the absence of definition and full implementation of the pilot approach (content, research questions, phases) and of a sound M&E system for documenting the lessons.
  • Achievement of improvements to the quality and cost-effectiveness of primary health care services is substantial. Project rehabilitation and equipment of SVPs culminated in improved availability of key primary health care services. The population's appreciation of these services (a proxy for quality) is evident both in the dramatic increases in use of services offered (prenatal services, vaccination rates) and in the results of a survey. There is no information on equity of access to these services within the oblasts. However, health promotion activities never got off the ground.
  • Achievement of the goal to develop a new cadre of primary health medical personnel through short-term training is modest. Primary care physicians have been trained and are now practicing as general practitioners, contributing to improvements in staffing of SVPs. Institutionalization of this training and legal recognition of this specialty were not achieved before the project's close, but are receiving support from a follow-on operation. Project investments in universal nurses training has failed to produce an appropriately trained nursing workforce for family medicine and general practice. The curriculum is not commensurate with the requirements of this nursing specialty and the training is not attracting capable candidates.
  • Achievement of improvements to the management and financing of primary health care services is substantial. With the availability and orientation of primary health care services in the pilot oblasts there is evidence of trends of a decrease in hospital admissions and an increase in outpatient care, in line with policy reform. The project introduced a financing, management and information system with per capita budget allocations and the pooling of funds at the local level. Some 637 SVPs were established as independent legal entities with their own bank accounts, giving them more autonomy in the management and financing of services. Investments in equipment, training, and systems helped develop management capacity.
  • Achievement of strengthened MoH capacity to evaluate and disseminate the results of the pilots is negligible. The generatation and use of data to document and disseminate lessons from the pilot experience fell short of expectations. MoH was slow in developing a baseline. The Government did not actively collaborate on sharing and using data and information from other sources. The 2002 DHS was embargoed for two years before the data became publicly available. A mid-term facility survey was only conducted at the end of the project. The ICR does not report on any dissemination activities.

5. Efficiency:

Project efficiency is modest. The design targets the highest priority diseases and most vulnerable populations through a package of cost-effective interventions and the decentralization of financing and decision-making authority to local-level services. However, efficiency was undermined by the lack of an adequate M&E system, which is an essential management tool for tracking and finetuning project performance and outcome and for promoting efficiency gains. Because of government bureaucracy, which is not accommodating of coordination across entities, project oversight and support from Government agencies were uncoordinated and slow. Ministry of Health was somewhat removed from the project implementation unit and capacity building of this unit was thus not directly beneficial to MoH. Procurement was undermined by heavy bureaucracy, processes, and rules and regulations.
6. M&E Design, Implementation, & Utilization:

Design. The objectives and indicators for the regional component of the project as presented in the project logframe do not underpin the operational learning intended in the pilot design. The goal of learning from a pilot approach (defining what was to be tested/learned) is not articulated in the logframe nor are there indicators to track such learning. The national objective is articulated around strengthening evaluation capacity and not around learning, finetuning and replicating. Not all indicators were commensurate with stated objectives and baselines were not collected. The pilot evaluation is briefly defined in three bullets in the logframe in terms of outcomes (quality, cost-effectiveness and efficiency; health status; users and providers' satisfaction), but the measures of how to get the best outcomes, the goal of the pilot, are not specified. Conditionality for M&E is reflective of a traditional project, calling for annual reports on indicators and a mid-term review. Implementation. Inadequacies of M&E design were never fully addressed during project implementation and are a reflection of the generally dificult and unsatisfactory situation with regard to data collection, interpretation, evaluation and publication in Uzbekistan. Soviet-style data collection is still pervasive, lacking consistency checks and critical assessment. The ICR notes caveats in the reliability of outcome data provided by the Borrower in its section. Annex 1 provides very limited data on outcome, both in terms of timeframe (baseline was only established in the second-to-last year of implementation) and in terms of scope (only a small fraction of original indicators are reported on).
Utilization. Because of the weak design and implementation of M&E under the project, both the quantity and quality of data were lacking and therefore not useful for informing decision-making.
7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative):

An unintended outcome of this project was Government's use of savings generated through the rationalization/consolidation of health facilities. Rather than allocate these savings to primary health, as originally intended, a portion of these resources were channeled to emergency hospital services.

8. Ratings:
ICR
OED Review
Reason for Disagreement/Comments
Outcome: 
SatisfactoryModerately SatisfactoryLack of rigor in designing and implementing a pilot approach and absence of good M&E design and implementation undermined the overall goal of documenting and applying lessons for finetuning and generalizing reform. Inappropriate indicators and paucity of relevant baseline and end-of-project data make it difficult to document project success although some improvements in coverage and efficiency are noted. Project investments failed to produce adequately trained nurses and to attract a sufficient amount of nursing students for staffing new primary health care facilities.
Institutional Dev.: 
ModestModest
Sustainability: 
LikelyUnlikelyGovernment not committing necessary resources for primary care reforms; use of savings for emergency hospitals; government not on board with downsizing facility norms for greater affordability; maintenance and supply of facilities not assured; public information campaign and public health capacity building not followed through; nursing outputs not qualified to take on PHC responsibilities in new facilities.
Bank Perf.: 
SatisfactoryUnsatisfactoryThe Bank neglected M&E during design and implementation. The documentation of learning was critical for this project both because it was a first investment in the sector and because its pilot design was supposed to support and document a learning process. This review also concurs with the ICR text (p. 4) showing that quality at entry was mixed. Supervision missions were not sufficiently frequent (less than twice a year) given the country context, the fact that this was a first health operation in the country (Bank and Government inexperience), the pilot nature of the operation, and the apparent weak Government commitment on certain aspects of health reform.
Borrower Perf.: 
SatisfactorySatisfactory
Quality of ICR: 
Satisfactory

NOTES:
- When insufficient information is provided by the Bank for OED to arrive at a clear rating, OED will downgrade the relevant ratings as warranted beginning July 1, 2006.
- ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness.

9. Lessons:

Pilot approaches that are not well defined (operational/research questions to be tested, roles and responsibilities, timing and content of discrete phases) or underpinned by a good M&E framework will compromise opportunities for learning and replication.
  • Project management functions that are not sufficiently mainstreamed into relevent line ministries will undermine efforts to build sustainable capacity and experience of the agencies with permanent responsibilities for sector management and oversight.
  • Organizational weaknesses in the government apparatus and public administration can negatively impact project effectiveness and sustainability.

10. Assessment Recommended?  Yes

          Why?  This is a first health sector operation in the country and it involves issues and challenges that would likely be relevant to those faced by many other countries/regions: the importance of M&E, especially in the context of a pilot design; the political economy of health sector reform; and decentralization and autonomy of health services delivery and management.

11. Comments on Quality of ICR:

The quality of the ICR is satisfactory overall. It is candid in discussing important shortfalls in project performance and outcome, as well as remaining issues and challenges. However, the ICR does not sufficiently describe or analyze the pilot implementation, its evaluation process, findings and lessons. In the absence of a good project monitoroing and evaluation system and the consequent lack of data, it has not attempted to exploit other data sources cited in the report to try and shed light on outcomes. Annex 1 reports on a small portion of original indictors. The use of proxies where possible and/or simply noting "data not available" might have allowed for a more complete treatment of original indicators.
Appraisal cost data shown in ICR (Annex 2) does not match PAD data (total estimated cost broken down by component; and cofinancing).

(ES-Rev4-Jul/05)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions