Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Second Health Project


  
1. Project Data:   
ICR Review Date Posted:
02/24/2006   
PROJ ID:
P009076
Appraisal
Actual
Project Name:
Second Health Project
Project Costs(US $M)
 200.0  131.72
Country:
Turkey
Loan/Credit (US $M)
 150.0  112.48
Sector, Major Sect.:
Central government administration, Health,
Law and justice and public administration; Health and other social services
Cofinancing (US $M)
 0  0
L/C Number:
L3802      
   
Board Approval (FY)
  95
Partners involved
 
Closing Date
12/31/2001 12/31/2004
         
Evaluator: Panel Reviewer: Division Manager: Division:  
Denise A. Vaillancourt
Roy Gilbert Alain A. Barbu IEGSG

2. Project Objectives and Components:

a. Objectives
As stated in the SAR, the primary goal of the Second Health Project is to reduce the burden of disease due to avoidable illness and disability in Turkey. To achieve this goal, the specific objectives of the project are to (a) improve equity of access to essential health services in 23 eastern, low-income, priority provinces; and (b) improve the quality of health care management in selected institutions.

Notes:

  • These objectives were not well stated in the Loan Agreement which only provided a summary statement of components, "The objectives of the Project are to (a) upgrade and expand the provision of primary health care in the Project Provinces; and (b) improve health care management."
  • In 1999 objective (a) (as stated in the SAR) was amended to read, as follows, "(a) improve equity of access to essential health services in 23 eastern, low-income, priority provinces and in five provinces affected by the August and November 1999 earthquakes."

b. Components (or Key Conditions in the case of Adjustment Loans):
The project provinces were to receive support under two components as follows. For component A. planned vs. actual cost estimates are provided in parenthesis. Component B was restructured and so here is presented twice, in its original conception with original cost estimates, and as restructured with actual total costs. Component A. Primary Health Care (US$155.6 million planned; US$80.58 million actual)

  • Primary Health Care Training (US$16.2 million planned; US$12.7 million actual): upgrade training of primary health care personnel in the project provinces through curriculum development, materials, revision of evaluation criteria, training of trainers, strengthening of inservice training, supervision mechanisms;
  • Basic Health Care Interventions (US$2.9 million planned; US$7.4 million actual): improve the effectiveness of essential health services through: extension of successful interventions to project provinces; public education campaigns, development of referral system, community and NGO involvement and operational research on cost-effectiveness of interventions; and
  • Infrastructure Development (US$136.5 million planned; US$60.48 actual): improve access to services in project provinces through construction, furnishing and equipping of urban inservice clinical training centers, urban health centers, rural health centers, public health laboratories and cold storages for vaccines, renovation of mid-sized hospitals.
Component B. Health Policy and Management (as originally conceived) (US$44.4 million planned):
  • Health Services Management Training (US$10.7 million): upgrading the training of staff in policymaking, planning, management, monitoring and evaluation;
  • Hospital Management Restructuring (US$4.5 million): decentralizing and restructuring management responsibility in six pilot hospitals;
  • Computerized Management Support Systems (US$12.4 million): introducing basic management support systems in Provincial Health Directorates and selected hospitals in the project provinces as well as evaluating a more comprehensive management information system in one urban pilot hospital in western Turkey; and
  • Pre-investment Studies and Project Management (US$16.8 million: conducting pre-investment studies to support a broader reform of the health sector, including burden of disease and cost-effectiveness studies.
Component B. Health Policy and Management (as revised in 1999) (US$42.2 million actual cost):
  • Health Services and Hospital Management (US$8.22 million): improving management capacities of provincial health managers and hospital managers;
  • Computerized Management and Support Systems (US$11.3 million): improving the cost-effectiveness and efficiency of health services;
  • Studies (US$5.5 million): analyzing burden of disease, health accounts and capacity building; and
  • Project Management (US$17.2 million).

Revisions to the project in 1999 were documented in amendments to the Loan Agreement but were considered minor and not presented to the Board. Revisions included: (a) a reduction in project size (fewer activities, lower cost -- essentially a drastic reduction in health infrastructure, dropping of referral system strengthening and community/NGO support) and a reallocation of US$22.5 million of the loan proceeds to earthquake-related activities; (b) restructuring the PCU through a substantial reduction in staff and a reorientation of its efforts back to its original mandate of coordination (vs. implementation); and (c) substantive corrections to the Management Service Agreement between MoH and UNDP/UNOPS for more rigor in financial management and auditing and a more narrowly defined mandate.

c. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Costs: The loan was made up of various currencies and the loan amount was expressed in US$ (not SDR) in the Loan Agreement. The project's loan accounts were first held in US dollars, then changed to Deutsche Marks and later to Euros. Planned vs. actual cost and financial data must thus take into account the various currency changes that the loan account was subject to as well as changes in exchange rates over the life of the project. Above-reported data on original vs. disbursed loan amount is derived from the loan agreement and the ICR's calculations. The Loan Department's data base show an original loan amount (US$154.21 million) and actual disbursements (US$111.75 million) that differ slightly from those reported in the ICR.

Reasons that actual disbursements were considerably less than anticipated include: slow start-up and subsequent dropping of activities (especially infrastructure); lack of procurement capacity; and the earthquake. Borrower contribution was adversely affected by the financial crisis which occurred during the project's life as well as to the need to reallocate public funds to earthquake emergency assistance. Dates: The original date was extended three times and the project closed on December 31, 2004, three years after its original closing date, in order to ensure completion of (delayed) infrastructure activities.


3. Relevance of Objectives & Design:

The project objectives were relevant from a number of fronts. Investments to improve health status targeted to low-income, priority provinces, along with efforts to improve sector efficiency, effectiveness and equity, were highly relevant both to the 1993 CAS as well as to the 2003 CAS. Objectives were also shaped by the seminal World Development Report 1993 Investing in Health, which advocated burden of disease analysis and delivery of an essential health care package for greater cost-effectiveness of investments. However, the objectives were overly ambitious given the relatively short timeframe of the project and the level of sector capacity in Turkey.

The design was flawed in that it was not underpinned with a coherent results framework. Expected outcomes and impact of the project were not sufficiently linked to proposed inputs and outputs. A good feature of this design was its effort to pave the way for health sector reform (rather than implement sector reform right away). It provided for studies that would inform and facilitate reform, most notably: burden of disease analysis and national health accounts.

4. Achievement of Objectives (Efficacy) :

Despite weaknesses in M&E (see Section 6), the project performance is rated as follows.

Achievement of the objective to reduce the burden of disease was negligible.
  • Between 1993 and 2003 overall infant mortality in Turkey dropped from about 53 to 29 per thousand (a drop of 45%). IMR in the project provinces, already lower than the country average, dropped by a smaller percentage (about 33%) falling short of the 40% target.
  • The goal to reduce maternal mortality by 30% in the project provinces cannot be measured in the absence of data.
  • Total fertility rate in the project provinces fell by 17%, falling short of the 30% target.
  • No data were available to measure progress against the goal to reduce infections due to tetanus, polio and measles by 95%.
  • There was a 79% increase in the use of antenatal care (ANC) services, but no data were available to measure the number of ANC visits per client against the project goal of 3.
  • The contraceptive prevalence rate inn project provinces increased by 37% exceeding the target of 30%.
  • Immunization rates against childhood diseases decreased from 65% to 54%, falling far short of the target to achieve a rate of 95%.
  • No data were available to measure other service performance targets and indicators: 30% reduction in child deaths due to acute respiratory disease, staffing rates of 95% for ambulatory health care; and 100% availability of antibiotics.
The specific objective of improving equity in access was not measured, but all indications are that it was modestly achieved.
  • Investments in basic health care were key in supporting immunization campaigns, family planning efforts, improvements to drug protocols, training of service providers and public information campaigns, and are assumed to have contributed to strengthening of these services in the project provinces. But health data shown in the ICR broken down by region indicate that inequities have persisted, and in some cases gotten worse.
  • Physical access might have been slightly improved through investments in the construction of training health centers, equipment of state hospitals and maternity hospitals. However, infrastructure planned under the project was dramatically reduced, and the extent to which health personnel trained under the project are working in the project provinces is not known.
  • It is likely that some positive contributions in improving physical access to essential health services in the five provinces affected by the earthquakes will be attributed to project support. Such an assessment will be carried out in 2006 when the Marmara Emergency Reconstruction Project (following the earthquakes) will be closed and evaluated.
  • Neither in the design nor in the evaluation were other dimensions of access fully assessed.
The objective of improving the quality of health care management is difficult to measure because the SAR did not provide sufficiently explicit indicators for this objective, nor did it collect baseline data to do so. ICR findings indicate that achievement of this objective was negligible.
  • Health sector management reforms did not materialize, undermining the full exploitation of the benefits of management training in which the project heavily invested.
  • Technical training is likely to have better prepared new health care workers for service provision in the provinces, but evaluation was limited to trainees' satisfaction, and it is not clear to what extent trained staff were returning or being assigned to the project provinces.
  • Improvements in the competence and cost-effectiveness of health services expected from investments in computerized management support system have yet to be realized.
  • Nevertheless, the project supported two key studies (National Burden of Disease Study and National Health Accounts), which are instrumental in informing decision-making of Government/MoH in the strategic planning and management of the sector.

5. Efficiency:

The project was inefficient on a number of fronts. The PCU was initially intended to carry out a coordination function, but took on an implementation role and its staff grew to over 200. This was an inefficient use of the MoH, which had the mandate for implementation, nor did it enable the capacity building of MoH through applied work. Furthermore, the PCU was supported by considerable technical assistance, which undermined opportunities for on-the-job training. This is especially so given that this was a second health investment, and capacity is reported to have been built under the first operation. The management services contract between MoH and UNDP was not initially subject to an independent audit and there is some risk that funds might not have been most efficiently used, and roles and responsibilities might not have been optimally distributed in line with comparative advantages. A review and revision of this arrangement, both for greater transparency and for a narrowing of UNDP's role, improved the situation somewhat.
The absence of a coherent results framework did not ensure a strong link between what the project was financing in terms of inputs and outputs, on the one hand, and its goals and targets, on the other. (See M&E discussion in Section 6. below.)
6. M&E Design, Implementation, & Utilization:

Design: The results framework was weak, revealing a poor link between project components and the ambitious development objectives. It would have been significantly strengthened with analysis of underlying factors behind low access, demand and behaviors. The results framework does not draw on knowledge, research, information that would enlighten and enable the goal of improving community and NGO involvement. Some aspects of the M&E design were impressive. Outcome (health status) and intermediate outcome (service delivery indicators) were defined, targets set (albeit ambitious) and baseline data were compiled for each of the 23 provinces. However, no indicators were defined or targets specified for equity of access and improved health care management objectives. Roles and responsibilities and modalities for the collection and analysis of data on indicators and for the overall management of M&E were not clearly defined, although the design documents do specify that a monitoring and evaluation committee would track indicators, project inputs and outputs, and oversee studies.

Implementation. The PCU did not ensure availability of staff that could monitor and evaluate project outputs and outcomes. Despite the availability of a project baseline and the multitude of data sources that could have been used to track progress, this remained one of the weakest components of the project. The mid-term review was a lost opportunity to bring to life a vibrant monitoring and evaluation system.
7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative):

Adherence was satisfactory to Section 4.02 of the Loan Agreement that required the Borrower to ensure that disposal of medical materials and waste and protection of X-ray facilities and staff would be undertaken in conformity with exigencies of the Borrower and the Bank.

8. Ratings:
ICR
ICR Review
Reason for Disagreement/Comments
Outcome: 
UnsatisfactoryUnsatisfactory
Institutional Dev.: 
ModestModest
Sustainability: 
UnlikelyUnlikely
Bank Perf.: 
UnsatisfactoryUnsatisfactory
Borrower Perf.: 
UnsatisfactoryUnsatisfactory
Quality of ICR: 
Satisfactory

NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness.

9. Lessons:

Deficiencies in initial project design can only be remedied midway through implementation if systematic and effective restructuring is undertaken.
  • A project's chances for success are reduced considerably when the statement of objectives are overly ambitious and their achievement depends on many other determinants, in addition to project inputs. A more coherent results framework during the design stage would establish stronger links in the results chain from inputs to outputs, outcomes, and impact.
  • Bank supervision tasks need to be more focused on achieving measurable outcomes rather than outputs alone.
  • M&E must become a focus of Bank work and teams have to be accountable for the formulation and supervision of realistic M&E tasks.

10. Assessment Recommended?  Yes

          Why?  Pursuit of lessons on political economy of reform, government commitment, restructuring, M&E and capacity building would inform many of the Bank's efforts to support health sector reform and to provide basic services to vulnerable populations and geographical areas.

11. Comments on Quality of ICR:

The ICR is of good quality overall, although it is much longer than provided for in the guidelines. It is very candid in its analysis, extensive in its efforts to provide evidence on the project's performance against its stated objectives and has put strong emphasis on the experience and lessons of project and program monitoring and evaluation. An important theme of this ICR is the importance of a project having a coherent results framework, both for facilitating rigorous monitoring and evaluation and for ensuring that project investments are well oriented around the achievement of project objectives.
A valiant effort is made in Annex 1 to report systematically on key performance indicators, in the absence of available data at the province level. Analysis of available data permitted the documentation of trends in some cases. However, some of the SAR targets were transcribed slightly differently in this ICR Annex. (e.g., reduce IMR by (instead of to) 40%. While correction of these differences would not have affected any changes in the project's ratings, care should be taken to record targets exactly as stated in the design document.

(ES-Rev4B-Dec/05)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions