Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Health 2 Project


  
1. Project Data:   
ICR Review Date Posted:
05/10/2013   
Country:
Uzbekistan
PROJ ID:
P051370
Appraisal
Actual
Project Name:
Health 2 Project
Project Costs(US $M)
 118.1  119.3
L/C Number:
C3979, CH124
Loan/Credit (US $M)
 40.0  41.2
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  09/09/2004
 
 
Closing Date
06/30/2010 03/01/2012
Sector(s):
Health (98%), Compulsory health finance (2%)
Theme(s):
Health system performance (33% - P) Tuberculosis (17% - S) HIV/AIDS (17% - S) Nutrition and food security (17% - S) Child health (16% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Erik A. Bloom
Judyth L. Twigg Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The project's development objective (PDO), as laid out in the Project Appraisal Document (PAD), was "to improve the quality and overall cost-effectiveness of health care services in Uzbekistan" (Section A.1 and Annex 1 of the PAD).
    The same PDO is used in the legal agreement. The PDO remained the same throughout project implementation.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

The project consisted of four components that were maintained throughout the life of the project.

Component 1. Primary Health Care Development. (Appraisal estimate of US$74.2 million or 82 percent of total project cost. Actual amount of US$101.4 million.) This component was to support investment in primary health care throughout the country, focusing on equipment, drugs, supplies, civil works, and training. This included infrastructure and equipment for rural primary health clinics not covered in a previous health project (Uzbekistan Health Project, US$30.0 million, 1998-2004), the improved provision of medications, the development of an urban primary health clinic, improved relevance of training programs and laboratory capacity, and creation of medical research and training centers.
Component 2. Financing and Management Reforms. (Appraisal estimate of US$4.3 million or 5 percent of total project cost. Actual amount of US$2.5 million.) This component was to support financial reforms, information systems, and planning capacity. This included support for the implementation of rural financing reform, support for an urban financing pilot, implementation of provider payment and management systems at the secondary level, and support for health sector management at the national level.
Component 3. Improving Public Health Services. (Appraisal estimate of US$5.9 million or 6 percent of total project cost, Actual amount of $11.4 million.) This component was to support capacity building for development of public health strategy, supporting the public health system, and scaling up activities to prevent communicable diseases.
Component 4. Project Management, Procurement, Monitoring, and Evaluation. (Appraisal estimate of US$2.4 million or 3 percent of total project cost. Actual amount of US$4.0 million.) This component was to support project administration and the monitoring and evaluation (M&E) system as well as support to improve procurement practices in the health sector.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

World Bank Financing

Total financing from the World Bank was US$41.18 million (US$40.60 million credit and US$0.58 million grant), compared to a US$40 million approved amount (US$39.48 million credit and US$0.52 million grant). The difference in financing is due to changes in the exchange rate (Annex 1 of the ICR). The project included US$4.2 million in unallocated resources (financed by IDA) and US$27.2 million in physical and price contingencies. These resources were fully allocated.

Government Contribution

At appraisal, the government was to finance approximately two-thirds of the project's costs (66.2 percent) and provide support to all four components. The largest single item with government financing was support for civil works, under component 1. The government provided 100 percent of the committed amount (US$ 78.1 million), accounting for 65.4 percent of total expenditure.

Project Costs

There were several adjustments in spending by component.

Component 1 saw an increase from US$97.9 million (appraisal amount) to US$101.4 million (final amount), due to increases in procurement costs. This was largely financed by unallocated expenses.

Component 2 saw a decrease from US$5.4 million to US$2.5 million due to savings in training costs.

Component 3 saw an increase from US$7.4 million to US$11.4 million, because of additional procurement due to growth in the iron-supplementation program and the inclusion of more health clinics. This was largely paid for through reallocation from other components.

Component 4 saw an increase from US$3.2 million to US$4.0 million due to the extension of the project.

In terms of categories, there was a significant increase in spending on civil works (from US$43.7 million to US$53.1 million) due to an increase in construction. This was financed by a reduction of spending on goods (from US$40.1 million to US$34.6 million) due to a reduction of training costs. The spending on consultants also declined (from US$ 4.1 million to US$ 2.0 million) due to the availability of financing from another partner, outside of the project.

Dates

The project was appraised on February 6, 2004 and approved on September 9, 2004. It became effective on December 20, 2004, three and half months after approval. The mid-term review was conducted on May 14, 2007, and the project was restructured on May 14, 2010 (including a 12 month extension) and on June 11, 2011 (including a 6 month extension). The first restructuring added procurement of dental equipment, technical assistance for a medical equipment maintenance system, and fine-tuning of the results framework. The second restructuring reallocated funds toward procurement of information technology equipment to develop the management information system. Both extensions of the closing date were to allow extra time for completion of implementation. The project was closed on December 31, 2011, eighteen months after the original closing date.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Relevance of objectives is rated high. At appraisal, the project was consistent with the World Bank's Country Assistance Strategy (2002-2004) and its objective "to enhance the efficiency of resource allocation and use in social infrastructure."

The project remained highly relevant as economic growth picked up and the health system shifted its focus to the secondary health system, as outlined in the current Country Partnership Strategy, 2012-2015 (p. 20). In particular, the Strategy calls on the World Bank to "....extend and deepen reforms, including the implementation of... [the government's] Public Health Strategy." The Public Health Strategy was supported by the project. In terms of goals for the health sector, the World Bank's strategy focuses on improving quality and access while ensuring financial sustainability, which would require improvements in cost-effectiveness as specified in the project's objectives. The Strategy also continues the focus on rural areas. At the same time, the project laid the foundation for the Bank's support to the government's focus on higher quality and improvements throughout the health sector (pp. 10, 12).

b. Relevance of Design:

Relevance of design is rated Substantial. The project was designed to improve the quality and cost-effectiveness of health services. To promote the quality of health care, design included the improved provision of medicines, training of medical staff, and the provision of new equipment (largely to be provided through component 1). At the same time, project design supported actions to improve the cost-effectiveness of health services, in particular through better training of medical personnel (component 1), management reforms and rationalization of hospitals (component 2), and improving public health services (component 3). Many of these interventions are complementary. For example, the training of medical personnel will lead to better health outcomes (quality) and to improve the primary health care’s system ability to treat patients efficiently (cost-effectiveness).

On the quality side, the provision of equipment and trained personnel at the primary level was expected to lead to better equipped health clinics, in turn leading to improved health care quality. Many of the same investments that were to lead to increased quality would also presumably lead to increased demand for primary health care, an important aspect of improving the overall cost effectiveness of health care. The project was designed to encourage greater use of preventive services (such as vaccination) as well as improving the public health system's monitoring, which are also associated with more cost-effective health outcomes.


4. Achievement of Objectives (Efficacy) :


(All baselines are from September 2004 and actual values from December 2011, unless otherwise specified.)

Attribution and coverage. The project was national in scope, with a particular focus on infrastructure and equipment in rural areas that were not covered in the previous health project. The project was developed in close collaboration with the Asian Development Bank (ADB), with planning to ensure that both projects complemented each other without overlap. Likewise, the project worked closely with USAID (on component 2) and JICA (on component 1) and took steps to ensure that the effort of these development partners complemented the project.

Data quality. Outcomes are measured using data and indicators from a number of different sources. These include professionally administered surveys supported by the project (in 2007 and 2011). Government data were also used, and the quality of the government's health information system improved towards the end of the project. A survey in 2007 was intended to validate much of the baseline data provided by the health information system at appraisal.

Improve the quality of health care services is rated Substantial.

Outputs

The project provided training and capacity building at the primary level, leading to 92.5 percent of primary clinics having a trained physician, a substantial increase from the baseline of 27.5 percent and exceeding the target of 70 percent. 3770 GPs were trained, exceeding the target of 2700. The percentage of physicians at the primary level trained or retrained as general physicians increased from 19.2 to 74.9, exceeding the target of 58 percent. The training package included upgrading knowledge on maternal health, which was done in coordination with an ADB-financed project. As a result of project support, virtually all primary health clinics have staff trained in the maintenance of medical equipment. In addition to supporting the cost of training, the project also provided support (in the form of equipment and supplies) to the training centers so that the training can continue after the project closes. The Bank reviewed the condition and quality of the equipment. The project team added that these reviews show that equipment is generally in use and well maintained.

The project also supported the provision of equipment at the primary level and the refurbishment of health clinics. 2,371 primary health clinics and 661 other facilities were constructed or rehabilitated. The project also provided a complete package of medical and laboratory equipment for primary health clinics in 13 regions, in coordination with the ADB-financed project. When combined with a previous Bank-financed project, this covered virtually the entire country. This led to an increase of the share of rural primary health clinics with electrocardiographs (from 97 percent in 2007 to 100 percent in 2011), autoclaves (from 44 percent to 97 percent), photoelectric colorimeters for laboratories (from 79 percent to 97 percent), and dry air sterilizers (69 percent to 83 percent). From a baseline of no dental equipment at local and regional clinics, 7.3 percent of local clinics and 100 percent of polyclinics were provided with dental equipment.

The project supported the adoption of international best-practice DOTS methodology for tuberculosis, including training to health care providers, support for laboratories, and the provision of drugs. The project also supported policy initiatives to focus on HIV/AIDS and tuberculosis, including training for medical professionals, sentinel surveys, and public awareness campaigns. The project increased the coverage of HIV/AIDS prevention activities (delivery of disposable syringes, voluntary counseling and testing, condom distribution, education materials, and training of community leaders) to groups at risk from 0 percent to 14 percent (risk groups included injections drug users, commercial sex workers, and men having sex with men).

Outcomes

Project surveys of 90 rural health clinics in 2007 and 70 in 2011 (ICR, pp. 49-50) report improvements in physician and nurse practices as the result of training. General practice physicians improved in the areas of newborn examinations, infant growth and development monitoring, nutritional consultation, and breast and cervical screenings. Nurses improved in the areas of awareness of signs of ectopic pregnancy and other abnormal conditions. Audits of patient charts indicate that monitoring of fetal growth has approximately doubled (from around 30 percent to 65 percent), HIV testing of pregnant women has increased fourfold, and screening of pregnant women for alpha-fetoprotein has increased by 4.5 times. There was a "significant" increase in the share of medical personnel with correct management of pregnant women with anemia, administration of iron preparations, and consultation on nutrition issues. While there is still room for improvement (18% of physicians were not trained in reproductive health), the progress noted in this area was substantial.

The percentage of essential medications stocked at primary health care facilities increased from an average of 38.9 percent stocked in 2004 to 64 percent in 2011. While few clinics stock more than 75 percent of essential medications, this is an improvement.

In the area of preventive care, there was an increase in early antenatal examinations (increasing from 77 percent to 87 percent in the first trimester, compared to a target of 85 percent) as well as near universal iron supplementation for target groups in four regions (increasing from a baseline of no coverage to coverage exceeding 95 percent in each of the four regions, exceeding the target of 90 percent). The improvements in antenatal care are likely to be due to an increase in income and public awareness campaigns (supported by ADB) as well as the project's efforts to improve the quality and effectiveness of public services.

While the project supported many efforts to prevent HIV/AIDS and to effectively treat tuberculosis, there is no clear accounting of the impact of these initiatives on health outcomes. The World Health Organization database shows a significant reduction (on the order of 50 to 70 percent) in mortality due to TB as well as the prevalence and incidence of TB from 2005 to 2011. The case detection rate was estimated to have doubled in the same period. Given the support for DOTS (which is considered the best approach to treating the disease) as well as the support provided to training and laboratory services and the importance of the public sector in controlling TB, it is quite likely that the project contributed to this impact. There are little reliable data on HIV/AIDS in Uzbekistan.

Project surveys suggest that around 95 percent of households are satisfied with the quality of primary health care compared to a baseline of 90 percent. Although the sample size is small (156 households), the data are indicative that the perception of quality remained high and may have increased. This is also consistent with the increase in demand for primary health care (rising from 3.8 visits per capita in 2004 to 4.7 visits per capita in 2011) and parallel decline in the usage of hospital care (see below). Other factors, however, may have contributed to change in demand, such as changes in pricing policies, morbidity, patients' ability to afford care, among others.

Improve the overall cost-effectiveness of health care services is rated Modest

Outputs

The project supported a number of efforts to improve the public health system of Uzbekistan. These included the training of 54 public health specialists (exceeding the target of 50), the establishment of a School of Public Health in the Tashkent Medical Academy and support to a public health department in another medical school, the development of an information system to monitor infectious diseases, and the approval of a public health strategy.

The project also supported efforts to reform health system financing, with an aim to provide incentives to providers and increase their autonomy. If applied properly, these incentives were to give providers greater possibility and incentive to respond to their area's specific health needs. At the same time, they would reward providers who are able to improve the efficiency of service delivery. To this end, with project support all rural and urban outpatient health clinics under the project converted to per capita financing (from a baseline of 86.2 percent of urban clinics and 21.3 percent of rural clinics in 2004). All 16 hospitals in the Central Rayon of Fergana Oblast participated in a case-based payment pilot, supported by the project.

The share of the government's spending on primary care increased from 41.7 percent in 2004 to 45.2 percent in 2011, which may reflect a change in the government's spending pattern. This change may have been brought about by the project’s focus on training and capacity building, support to primary health care (see discussion on quality above), and efforts to bring more incentives for cost-effective care into the health care system.

Outcomes

The project measured a number of typical indicators of cost-effectiveness, including the hospitalization rate of the rural population (dropping from 11.1 percent to 10.5 percent, compared to a target of 10 percent), reduction of hospital referrals (dropping from 20 percent to 12 percent, compared to a target of 15 percent; this indicator was formally dropped but provides insight into changing trends), and increase in the use of primary health care (cited above), from 3.8 visits per capita in 2004 to 4.7 visits per capita in 2011. While Uzbekistan's income improved during this period, it is unlikely that this on its own would lead to an increase to the change in utilization patterns. A pure income effect is likely to lead to an increase in demand for hospital services (given no change in the quality of primary health care) and private health services. Since the price of service has remained stable, this is likely to reflect a change in perception on the part of the beneficiary population.

The project's outputs and outcomes related to preventive services (see above) also contributed to achievement of cost-effectiveness of health care services.


5. Efficiency:


Efficiency is rated substantial. At the time of appraisal, the project team developed a detailed cost-effectiveness analysis. The analysis starts with the project's impact on reducing disability-adjusted life years (DALYs) lost due to disease, focusing on the gain in productivity due to reduced morbidity and mortality. The cost-effectiveness analysis includes the project's costs, including local costs, incremental costs, and the cost of the credit, as well as the benefits including savings to the health care system, the impact of public health interventions, and the gain in DALYs. Although the analysis appears to be well-developed, there is significant discrepancy in the reported economic rate of return. The PAD reports the project's economic rate of return as 4 percent (page 31) while Annex 4 reports the internal rate of return at 84 percent. As part of this ICR review, IEG estimates that the rate of return was 45 percent (using data that were provided in the PAD). The ICR provides no additional information on the estimated economic rate of return except to indicate that the economy grew at a faster rate than originally estimated. While no value is given, the high rate of economic growth (increasing the value of the intervention in economic terms) and the generally positive results seen in the project suggest a positive rate of return.

The project’s close collaboration with the Asian Development Bank shows good coordination and a division of work that would help the project focus on its objectives. The collaboration was to consist of upstream division of technical work and coordination of project responsibilities, as well as downstream collaboration during implementation, including coordination of activities and supervision. This led to a more efficient use of preparation and supervision funds.

The project also targeted its inputs carefully, focusing on geographic areas that were not previously covered. In addition, the coordination with the government and other partners allowed greater targeting of technical inputs as well. On the ground, government support to the project through the construction of a Training Center in 2008 led to an decrease from the planned cost of training, even though more physicians than planned were trained. There was a savings of US$ 2-3 million for facilities rental and procurement of training equipment.

The delay in project implementation contributed negatively to the efficient use of project resources, as it tied up the credit and required additional Bank resources to address bureaucratic issues. In particular, the decision to delay the request for the final six-month extension to the end of the original project complicated planning, when it should have been apparent previously that the project needed to be extended.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


The project's objectives were highly relevant, responding to the country program and the government's strategy, both at the time of approval and the closing of the project. The project's design was straightforward, with planned activities logically and plausibly leading to achievement of the objectives. Provision of equipment and training led to measured increases in physicians' and nurses' knowledge and improved clinical practices. There is less evidence that the project had a major impact on improving the cost-effectiveness of health services. Effective collaboration with the ADB, geographic targeting of inputs, and leveraging of government investments were evidence of substantial efficiency of allocation of project resources.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:


The government has shown ownership in the project, both by investing more than was originally agreed at the time of appraisal and by requesting a follow-on project from the World Bank, the Health Systems Improvement Project (approved on April 7, 2011 for US$93 million). The follow on project focuses on strengthening the health system, ensuring quality health care, rationalizing the hospital sector, and responding to non-communicable disease. These themes build on the Health 2 Project and are likely to support the investments already made. The government continues to support the areas covered in the project, while moving to improve hospital services.

The economy of Uzbekistan has grown quickly during the project period, providing substantial revenues to support the project achievements. The current government strategy supports the goals of the project (improving primary and preventative care), while moving towards more specialized issues covered in the new project.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:

Project preparation did not make realistic assessment of fiduciary issues. The project took advantage of the presence of other partners. While the approach to environmental safeguards reflected regional practices at the time, the project could have been more proactive in this respect.

Analysis of alternatives and risks

The project team analyzed several different options and alternatives. The original request from the government was to focus on hospital equipment, showing that there was significant discussion with the government on its priorities. Likewise, the joint work with the ADB also included significant discussion about options. The PAD outlined a list of lessons learned from the previous health project in Uzbekistan and other experiences. The PAD included an analysis of risks that was largely adequate but did not reflect the risk of government delays in procurement or the environmental risk of additional civil works.During preparation, there was a review of the lessons learned from the previous project (Uzbekistan Health Project; approved on September 22, 1998 for $30 million). This included the importance of timing training and the purchase of the equipment, as well as the importance of improving the public health system. The PAD justified public sector investment in terms of reaching the poor and reducing health shocks however it did not explicitly consider the externalities associated with investment in public health.

    Monitoring and Evaluation

    The M&E framework was unnecessarily complicated, with 15 outcome and 44 output indicators (pages iv to xv and page 13 of the ICR). There does not appear to have been an assessment of the government's capacity to monitor the results framework. See Section 10 for further assessment.

    Partnerships

    While the project did not have co-financiers, it was designed to work in close collaboration with several other health sector donors, including JICA, USAID, and ADB. among others. While a joint project may have been ideal, the parallel approach that was included in the project design was realistic given the time required to set up a joint structure and the nature of the other partners. The PAD had a detailed description of the expected division of labor between the ADB and the World Bank. The two Banks also worked closely in carrying out joint studies during the preparation period. While the two Banks collaborated closely, they agreed not to include any cross-conditionalities that could have delayed implementation.

    Safeguards

    The project was developed in accordance with regional standards, and the PAD outlines the steps taken to assess the environmental impact of the project. It was rated as C (no expected environmental risk) at the time of appraisal. This environmental assessment was carried out jointly with the ADB.

    Project Readiness

    While the legal agreement had nine dated covenants, none of them were conditions for readiness and the project was certified as being ready for implementation. The project became effective about three months after approval, which reflects the work done to ensure readiness.

    The financial management capacity of the implementing agency was judged to be adequate. The project did not establish a mechanism to capture the value of the government's in-kind contributions (for example, support to civil works). which formed an important part of project financing. The design did not take into account Uzbek procurement procedures and their potential impact on the project. In particular, Annex 6A of the PAD made no mention of local procurement requirements and their conflict with Bank procedures. Given that the conflict between national legislation and Bank policy must have impacted other projects, this is an important shortcoming in design.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

    The Bank team work diligently to implement the project. The team identified a number of issues with design and implementation, such as the monitoring and evaluation framework and the delays in procurement, and sought to address these issues quickly. There were a number of bureaucratic obstacles in implementing the project. These included delays in the approval of the two restructuring agreements. The Bank continued to press the government on these areas, and eventually the restructuring agreements were approved. In both cases, the Bank prepared the documentation well before it was needed. There were a number of disagreements between the government and the Bank over procurement, which led to new agreements to ensure a smoother implementation of the project.

    In the area of environmental safeguards, the Bank carried out several detailed reviews to confirm that the initial rating of C remained valid. Although it was not necessary to change the ratings, this review led to changes in practices and revised agreements with the government. This was important, given the government's decision to provide more resources for civil works.

    The Bank team also continued to work with other development partners and supported the training of key staff to speed up implementation. The Bank team had three task team leaders during the period of eight years, which is reasonable turnover. The change in project teams was relatively smooth, and there was continuity among the other team members.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The government had a strong sense of ownership as demonstrated by the high level of government contributions at the national and local level, as well as the government’s development and implementation of its health strategy. By the end of the project, the government had developed a cross-sectoral public health strategy that coordinates government activities in public health. One consistent problem was with government procedures, which were often inflexible, leading to delays in procedures (including procurement and the project restructuring). However these problems were more systemic and not confined to the project. Eventually, the government was able to resolve these issues.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

There were two implementing agencies responsible for the project. The Joint Project Implementation Bureau (originally known as the Central Project Implementation Bureau) was responsible for the day-to-day operation of the project. The Ministry of Health was responsible for providing technical guidance. The Joint Implementation Bureau was to report to the Ministry of Health.

Joint Project Implementation Bureau. Overall, the Joint Project Implementation Bureau provided the necessary support to ensure that the project advanced. Although audits were often qualified, this was largely for technical reasons (difficulties identifying the value of contributions of local governments), Most key fiduciary documents were submitted in a timely fashion, including the annual audit and quarterly financial reports. Initially, monitoring and evaluation were weak, and this situation was not fully addressed until the final years of the project. While the Bureau generally had necessary personnel, there were often gaps and turnover of key staff.

Ministry of Health. Initially the Ministry was not an active player in the project, leaving both the fiduciary and technical elements to the Ministry of Finance and the Joint Project Implementation Bureau. With time, the Ministry became more active in the project and provided increased technical support. The Ministry realized the project’s role in improving its stewardship of the health sector and harnessed capacity building from the project and other development partners. The Ministry adopted the project’s monitoring and evaluation framework and supported efforts to develop the new School of Public Health, to train public health specialists, and to introduce the health information system. The Ministry is also playing a more significant role in the follow-on project. Moderately satisfactory.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The PAD contained an extensive results framework, with outcome and impact indicators mapped to the project objectives and a a large number of output indicators mapped to each of the project's components. At the time of appraisal (in 2004), the project team identified 15 outcome indicators as well as 44 output indicators. Several of the outcome indicators were actually outputs (related more to activities than to the expected outcome). Baseline data were not available during project preparation. The indicators were designed to be collected from a planned Demographic and Health Survey, which was carried out in 2004. The Joint Project Implementation Bureau was responsible for the collection and analysis of data. At appraisal, the Bureau did not have the necessary M&E capacity.

b. M&E Implementation:

Initially, the government lacked the capacity to monitor the results framework. According to the ICR (footnote 2, page ix) only four of the output indicators were actively monitored prior to the mid-term review. A full set of baseline data at the outcome level was available by 2005, although the project was only able to collect data on a small set of output indicators. Since the planed Demographic and Health Survey was not carried out, the project organized a survey in 2007 that provided data on the effectiveness of health facilities. This survey was followed up by an end-of-project study in 2011.

During the mid-term review, the government and the Bank agreed to overhaul the indicators, developing a new results framework with 9 outcome and 15 output indicators. Although it took three years to formally adopt these indicators, they were used informally after the mid-term review. It was not until 2010 that the project had sufficient capacity to fully monitor the indicators. After this date, the project was able to monitor the full set of indicators, and the ICR reports both the new results framework and progress against the original outcome indicators.

a. M&E Utilization:

After the M&E system was firmly established with the new set of indicators, the government started using the project indicators both to monitor the project and for its broader monitoring of the health sector. The project's indicators have been incorporated into the national system and are still monitored after the project's close. They also formed the basis of the M&E system for the follow-on project.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

The project was classified as "C" under the environmental assessment. This followed regional standards at the time. The Bank continued to monitor the project's environmental standards and there were several reviews to confirm the environmental assessment. In particular, the Bank confirmed that the civil works were done in accordance to Bank guidelines. The Bank and government agreed that that further government construction would be implemented outside the project. The regional safeguard teams endorsed this approach.

b. Fiduciary Compliance:

While financial management reports and disbursement applications were normally prepared accurately and on time, the project generally received qualified audit reports. The auditors were largely concerned about the lack of documentation on government counterpart financing, which was largely provided in kind by local governments. Even when information was provided about the size of counterpart financing, the government did not provide sufficient documentation. This is a minor shortcoming in the project's fiduciary monitoring.

Government procurement procedures were often inflexible, leading to an initial delay in procurement activities during the project's first two years. While the Bank did not have to issue any significant procurement waivers, the procurement process was delayed by the government's requirement to carry a post-bid price verification of the procured goods. These issues were systemic and not confined to the project and in 2007, the government and Bank signed an MOU under which the government agreed to standard Bank requirements.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Negligible to Low
Negligible to Low
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The ICR identifies several lessons from the project (page 25). IEG observes two additional lessons from the preparation and implementation experience of the project:
    • Partnership can come in many forms and have a strong impact on achieving a project's objectives. As shown in the discussion on efficiency (section 5), the World Bank worked closely with the Asian Development Bank, allowing each agency to specialize in certain areas to support the government's health program. Although this was not formally a joint project, this cooperation had many of the same characteristics as a joint projects and gave each partners many of the same advantages.
    • At the same time, partnership carries risk. At the same time, as discussed in the ICR, informal partnerships have risks. One of the assumptions during project preparation was the availability of USAID-financed technical assistance. This support was cancelled after a dispute between the government and USAID.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is well prepared and provides a wealth of data on the both the original and revised indicators, including a detailed explanation of their achievement and their significance for the project. The ICR also includes a detailed summary of the project's activities, allowing the reader to understand what activities were actually carried out. The ICR provides a frank assessment of many elements of project implementation, including the delays in restructuring and the reassessment of the environmental classification.

Despite the general high quality of the ICR, one shortcoming is its narrow focus on indicators to judge the project's success. For example, table 10 (page 21) outlines a weighting system for rating the achievement of objectives. This can lead to false perceptions of a project's impact, by focusing on averages of indicators that often weigh both outcomes and outputs. The data contained in the ICR are substantial and should have permitted a more in-depth analysis of the causality chain.

One minor shortcoming in the ICR is its partial review of the economic analysis in the PAD.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions