Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Second Women's Health & Safe Motherhood


  
1. Project Data:   
ICR Review Date Posted:
04/25/2014   
Country:
Philippines
PROJ ID:
P079628
Appraisal
Actual
Project Name:
Second Women's Health & Safe Motherhood
Project Costs(US $M)
 38.0  n/a
L/C Number:
L7290
Loan/Credit (US $M)
 16.0  10.29
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
 0  0
Cofinanciers:
Board Approval Date
  04/21/2005
 
 
Closing Date
06/30/2012 06/30/2013
Sector(s):
Health (75%), Central government administration (15%), Sub-national government administration (5%), Compulsory health finance (3%), Other social services (2%)
Theme(s):
Population and reproductive health (29% - P) Health system performance (29% - P) HIV/AIDS (14% - S) Social safety nets (14% - S) Administrative and civil service reform (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Judith Hahn Gaubatz
Judyth L. Twigg Lourdes N. Pagaran IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Loan Agreement (page 25), the project objective was as follows:
      • To assist the Borrower in improving women's health by (i) demonstrating in selected sites a sustainable model of delivering cost-effective reproductive health services to disadvantaged women; and (ii) establishing support systems to facilitate country-wide replication of lessons learned within the framework of its Health Sector Reform Agenda.

    The objective was similarly articulated in the Project Appraisal Document (PAD, page 4): "To contribute to the national goal of improving women's health," although the means to achieving the objective ("by") were worded more specifically in the PAD: (i) demonstrating in selected sites a sustainable, cost-effective model of delivering health services that increases access of disadvantaged women to acceptable and high-quality reproductive health services and enables them to safely attain their desired spacing and number of children; and (ii) establishing the core knowledge base and support systems that can facilitate country-wide replication of the project experience as part of mainstream approaches to reproductive health care within the framework of the Health Sector Reform Agenda.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

1. Local Delivery of the Integrated Women's Health and Safe Motherhood Service Package (Appraisal: US$ 13.4 million; Actual: n/a): This component aimed to support local government units in delivering an integrated health service package to women. The package was to include maternal care, family planning, and sexually-transmitted infections control, and it particularly was to focus on encouraging facility-based, versus home-based, deliveries for pregnant women. The activities were to be initiated as a pilot project in six project sites. Activities were to include: establishment of women's health teams and obstetric care teams; pilot interventions for reaching groups at high risk of contracting sexually-transmitted infections (i.e. commercial sex workers and returning overseas workers); establishment of efficient procurement and logistics management systems for drugs and contraceptives; safe blood supply; and provision of performance-based grants to encourage (i) facility-based deliveries (awarded to pregnant women and women's health teams); and (ii) enrollment of the poor in the subsidized national health insurance program (awarded to local government units).

2. National Capacity to Sustain Women's Health and Safe Motherhood Services (Appraisal: US$ 2.5 million; Actual: n/a): This component aimed to develop capacity of the Department of Health (DOH) to manage health service delivery by the various local government units. It also intended to prepare the DOH to eventually scale up successful interventions throughout the country. Activities included: establishment of operational and regulatory guidelines for the service package; training to service providers; research; and monitoring and evaluation.

The following activities were modified at the time of project restructuring in 2010:

    • Sub-project financing of US$ 1.5 million to local government units for civil works and goods (i.e. for health unit upgrades) was dropped.
    • Training facilities were expanded to national scope (although the allocation was not increased).
    • Allocation for performance-based grants increased from US$ 3.3 million to US$ 5.0 million.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost:

    • The appraised project cost was US$38.0 million. The ICR does not provide the actual project cost, although it indicates that the final cost was lower than planned. The following factors appear to have contributed to the shortfall in spending: the intensification of government focus on maternal health (therefore activities related to contraception and sexually-transmitted infections control were not fully implemented); securing of external funding for civil works (therefore sub-project financing for civil works was dropped); incomplete implementation of operations research and M&E activities; and lower-than-expected disbursement of performance-based grants.

Financing:
    • The project was to be financed in part by a Bank loan of US$ 16.0 million. Due to shortfalls in implementation, the actual loan disbursed US$ 10.7 million. US$5.28 million of the remaining loan amount was cancelled.
    • In 2009, external sources of financing became available to local government units for civil works and equipment (i.e. health facility upgrades). These sources included the national budget and Spanish Aid Agency grant funding. Therefore, the project's original allocation of US$ 1.5 million for goods and civil works for public health units was re-allocated.
    • The project's allocation for performance-based grants was increased from US$ 3.3 million to US$ 5.0 million.

Borrower Contribution:
    • The Borrower provided US$ 16.0 million in counterpart funding, compared to the appraised amount of US$22.0 million. The shortfall was due to incomplete implementation of project activities.

Dates:
    • June 2010: The project was restructured to better support the government's intensified focus on reducing maternal mortality. Project activities focused on maternal health (training, performance-based grants, behavior change communications) were scaled up, while sub-project financing for civil works was dropped. Key indicators were refined to reflect these changes. The project closing date was extended from June 2012 to June 2013.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High. The project's objective was highly relevant to country conditions at the time of appraisal, where the maternal mortality rate was relatively high at 186 per 100,000 live births (2005) and the contraceptive prevalence rate relatively low at 49% (2003). The project is directly linked to the objectives of the government's national health sector framework, which includes the newly established policy to encourage all pregnant women to deliver in health facilities rather than at home (even with the aid of a skilled birth attendant). The objective is also highly consistent with the Bank's most recent Country Assistance Strategy for FY2010-2012, which identifies improved access to health services, including facility-based deliveries, as a key outcome. It is also highly relevant to the Bank's corporate priorities to achieve the Millennium Development Goals.

b. Relevance of Design:

Substantial. The main project interventions - supporting facility-based deliveries, access to contraception, and control of sexually-transmitted infections - were likely to lead to improved women's health. The project was intended as a pilot program, primarily to test the government's newly-established policy encouraging facility-based births. Project sites were selected in six provinces (out of a total 60), while institutional and capacity support would also be developed at the national level to prepare for a scale-up of interventions. The project specifically targeted poor and disadvantaged populations by incentivizing local government units to achieve enrollment targets for the subsidized national health insurance program.


4. Achievement of Objectives (Efficacy) :

To improve women's health

Modest. Although there were substantial achievements in increasing facility-based deliveries, there were shortcomings in the areas of contraceptive prevalence and control of sexually transmitted infections.

Demonstrating in selected sites a sustainable model of delivering cost-effective reproductive health services to disadvantaged women:

Outputs

    • Establishment of health teams to deliver the Women's Health and Safe Motherhood (WHSM) service package. This included women's health teams (community level), basic obstetric care teams, comprehensive/ emergency obstetric care teams, and itinerant teams.
    • Establishment of 29 training centers to provide training on maternal health. 100% of basic obstetric care teams in the project areas received training.
    • Information and education campaigns to encourage facility-based deliveries. The proportion of women (who had given birth in the past six months) who had stated birth plans increased from 65% in 2009 to 81% in 2013, achieving the target of 81%.
    • Issuance of ordinances on contraceptive self-reliance, in order to ensure a consistent supply of drugs and supplies. The proportion of local government units in project areas that passed an ordinance on contraceptive self-reliance increased from 36% in 2009 to 80% in 2013. This fell short of the target of 100%.
    • Establishment of local procurement systems in order to ensure efficient supply management. The proportion of health units in project areas that had not experienced stock outs of pills, injectables and contraceptive devices for the past 6 months increased from 16% in 2006 to 100% in 2013.
    • Provision of supplies for safe blood supply.
    • Awarding of performance-based grants to local government units to encourage enrollment of the poor in the subsidized national health insurance program. The proportion of local government units in project areas that sustained 75% enrollment rates for the insurance program increased from 81% in 2009 to 90% in 2013. The proportion of facility-based deliveries in project areas financed through the insurance program increased from 10% in 2009 to 38% in 2013. This fell slightly short of the target of 40%.
    • Awarding of performance-based grants to women's health teams and health facilities to encourage facility-based births.
    • Establishment of protocols for screening for sexually-transmitted infections. The proportion of clients who were properly screened increased from 59% in 2009 to 87% in 2013. This fell short of the target of 100%.

However, outreach to three high-risk groups (female sex workers, returning overseas workers, and young adults) for sexually transmitted infection control, and behavior change communications for contraceptive use, were not implemented as planned.

Establishing support systems to facilitate country-wide replication of lessons learned:

Outputs
    • Development of Women's Health and Safe Motherhood (WHSM) standards for use in accreditation criteria. The proportion of health units in project areas that were accredited for maternal health service delivery increased from 9% in 2006 to 86% in 2013. This fell short of the target of 100.0%.
    • Dissemination of WHSM service guidelines for local government units and local health teams.
    • Baseline survey for WHSM data in project areas.
    • Routine collection of WHSM data by local government units.

However, the endline surveys for WHSM data, operations research, and impact evaluation were not carried out as planned.


Overall Outcomes
Assessment of the project's achievements are based the intermediate outcomes reported below, which are considered proxy indicators for improved women's health. A longer time period and multiple variables would likely be needed to affect broader health outcome indicators such as maternal mortality rate and sexually-transmitted infection prevalence rate.

Maternal health
    • The proportion of births taking place in a health facility increased in the six project provinces from 42% in 2009 to 87% in 2013, surpassing the target of 80%. The proportion in the individual provinces ranged from 73% to 100%.

Contraceptive prevalence
    • The contraceptive prevalence rate in project areas increased from 35% in 2009 to 39.4% in 2013. This fell short of the target of 45%. The proportion in the individual provinces ranged from 31% to 50%.

Sexually transmitted infections
    • The proportion of clients who knew the three ways to prevent sexual transmission of HIV increased from 39% in 2009 to 55% in 2013. This fell short of the target of 70%.
    • There were no data reported on behavior change that would impact the prevalence of sexually-transmitted infections among women.

5. Efficiency:

Efficiency is rated Modest due to lack of sufficient evidence on cost-effective use of project resources. The ICR (page 18) reports that a "costing analysis" was conducted midway through the project period to identify the cost of expanding the pilot interventions across the country. According to the ICR, "this assessment had indicated that the pilot was cost effective and recommendations were made for scaling up the [project] to across the country in 2010," although the initial phase of the project had yet to be formally evaluated. However, specific data from this analysis were not reported in the ICR.

In addition, there were significant delays during the project period, including in finalizing the project restructuring, carrying out M&E surveys, procurement, and contracting out of project activities. These delays likely diminished the benefits realized from the project.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Relevance of the project's objectives is rated High, and relevance of the project's design is rated Substantial. Achievement of the objective to improve women's health is rated Modest due to shortcomings related to contraceptive prevalence and control of sexually transmitted infections, although there were substantial achievements in intermediate outcomes related to maternal health. Efficiency is rated Modest due to lack of sufficient information.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:


The government's adoption of facility-based deliveries as official policy will ensure implementation support for these activities, such as institutional arrangements, capacity building efforts, and direct linkages to the national health insurance program.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:

The project's design was overall technically sound, drawing from international experience in reducing maternal mortality rates in countries with similar socio-economic status. The pilot approach was intended to test the women's health care package in six provinces (out of a total of 80 provinces), learn lessons from implementation experience, and then scale up the interventions nationwide. The institutional arrangements drew on existing networks of public and private health providers and were consistent with the recent government policy to devolve service delivery to local government units (LGUs). However, capacity of LGUs to implement the activities was low, and capacity of the Department of Health to manage the multiple LGUs was also inadequate, particularly with regard to fiduciary transactions. These issues were identified in the risk assessment, which was comprehensive and realistic. However, some mitigation measures added further complexity to the project design (a "contraceptives-for-drugs" swap that did not work) and others were not completed in a timely manner (initial deposits by LGUs into newly opened project accounts). M&E design was adequate, although target figures, which were based on nationwide data rather than project-specific areas, were not updated as planned.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

The initial project implementation period was marked by very low disbursement rates. The Bank team introduced a Rapid Results Approach in 2009/10 to facilitate implementation, which resulted in a somewhat improved rate of disbursement. However, the government subsequently announced its decision to intensify focus on maternal health outcomes and scale up the facility-based interventions nationwide. This shift in policy, along with continued slow disbursement, led the Bank team to discuss the possibility of restructuring and partial loan cancellation. However, there were lengthy delays in finalizing the project restructuring, and the ICR (page 12) suggests that a more extensive project restructuring (beyond modifications to the key indicators and reallocation of funds) might have been more appropriate. The Bank team was proactive in addressing financial management problems, namely delays in processing performance-based grant disbursements; however, these matters were not fully resolved by project closing, and project funds were not fully utilized. M&E implementation also had shortcomings, as there were lengthy delays in carrying out the baseline and endline surveys, affecting the timeliness of data collection.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government was strongly committed to the overall project objective to improve women's health, including introducing the new facility-based deliveries strategy. Midway through the project period, in part due to the initial achievements of the project, the government moved forward with scaling up the facility-based approach nationwide, although a formal evaluation of the pilot had not yet been conducted. However, the scaled-up government program did not include the full range of activities included in the Bank project, such as family planning and sexually-transmitted infection control, and therefore these latter activities did not receive adequate attention for the remainder of the project period.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:

The Department of Health (DOH), as the primary implementing agency, was strongly committed to shifting from a home-based to facility-based delivery approach, using the Bank project as a pilot initiative. The DOH was effective in implementing critical project interventions, including accreditation of health units, establishment of health teams, and provision of grants for pregnant women and health teams. However, project management staff, including procurement and financial management specialists and monitoring and evaluation specialists, were not hired as planned due to procurement delays. The DOH also decided to carry out training and behavior change communication activities with its own staff rather than contract out to a consultancy as originally planned; this decision contributed to lengthy implementation delays. Fiduciary performance of local government units also had shortcomings, particularly with regard to creating budget lines for project interventions and reporting back on expenditures on performance-based grants. The latter contributed to difficulties in verifying results against performance-based grants.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

M&E indicators and arrangements were clearly identified in the project design. Baseline data were based on national figures and were to be updated for project-specific provinces when a baseline survey was conducted. Although the baseline survey was carried out, the targets were not updated as planned. At the project restructuring, the original indicators were modified to reflect the government's decision to focus on facility-based deliveries.

b. M&E Implementation:

The expected M&E specialist position was vacant for most of the project period. The planned endline survey, impact evaluation, and operations research were not carried out. There were no validation mechanisms for the performance-based grants, and therefore results could not be verified against grant payments.

a. M&E Utilization:

There is limited information on the use of M&E data to inform project decision-making, with the notable exception of the government's decision to scale up the maternal care interventions due to the observed increase in facility based deliveries in the initial two project provinces.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

The project was classified as a Category "B" project due to medical waste management and construction issues (noise generation and construction waste). Safeguards policies on Environmental Assessment (OP/BP 4.01), Involuntary Resettlement (OP/BP 4.12), and Indigenous Peoples (OP 4.10) were triggered. An Environmental Management Plan was developed and disseminated, which included guidelines on site management and waste management, with monitoring of compliance by the Department of Health. A Land Acquisition and Resettlement Policy Framework to address potential involuntary resettlement issues, and an Indigenous Peoples' Development Plan to engage indigenous communities, were also prepared.

Only minor civil works were carried out under the project, and the ICR (page 14) reports that there was general compliance with the environmental and waste management guidelines. The project also complied with the safeguard guidelines on Indigenous Peoples. The safeguard measures on Involuntary Resettlement did not, in fact, need to be invoked during the project period.

b. Fiduciary Compliance:

Financial management: There was mixed performance on financial management, due to delayed submission of reports and delays in preparation and submission of paperwork related to performance-based grant withdrawals and expenditures. Quarterly financial reports and annual audit statements were submitted regularly, albeit with some delays. Out of the seven audit reports, three had qualified opinions and one had an adverse opinion. Issues were as follows: (i) misclassification of accounts in 2006; (ii) understatement in cash balance and negative balance in the Other Payables account in 2007; (iii) foreign exchange loss and receivables from local government units in 2009; and (iv) unreconciled balance between client's statement of funds and Bank system. The ICR (page 15) reports that these issues were subsequently addressed by the implementing agency.

Procurement: Although there were lengthy delays in processing procurement contracts, there were no major problems reported in procurement.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Risk to Development Outcome:
Negligible to Low
Negligible to Low
 
Bank Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

Lessons drawn from the ICR (pages 23-24) and adapted by IEG:
    • A pilot project can be an effective approach for demonstrating results to various stakeholders, who may then become active participants in the program. In the case of this project, the initial achievements in facility-based deliveries led to increased budget allocations from the national government and grant support from an external donor.
    • Despite utilizing existing institutional structures, fiduciary capacity still needs to be carefully assessed as the project may introduce new procurement and financial mechanisms. In the case of this project, the decentralized government units and existing networks of private and public health providers were expected to implement the project interventions; however, the lack of experience with financial reporting led to significant implementation delays.
    • Performance-based grants can be effective mechanisms for linking demand-side and supply-side interventions to lead to desired outcomes. However, validation mechanisms need to be put in place and fiduciary capacity needs to be adequate to manage the financial incentives introduced by the performance-based mechanism. In the case of this project, the existing fiduciary systems were input-based and did not quickly adapt to more results-based financing approach.

14. Assessment Recommended?

Yes
Why?
To verify outcomes on maternal health and to learn lessons, particularly about the effectiveness of the performance-based grants.

15. Comments on Quality of ICR:


The ICR is overall satisfactory. It provides an detailed assessment of the factors, both positive and negative, that affected implementation. It draws useful lessons on implementing a project that seeks to introduce a significant policy change. However, the assessment of project outcomes is only marginally satisfactory. The ICR provides a limited analysis of data and outcomes; for example, Section 3.2 of the ICR only lists project indicators and identifies them as "achieved" or "not achieved," whereas an analysis linking project outputs to project outcomes (as evidenced, in part, by the project indicators) would have strengthened the assessment. The ICR reports that the Borrower's ICR was not yet completed at the time of submitting the Bank's ICR. Actual project cost is not reported.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions