Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - India: Reproductive & Child Health Second Phase


  
1. Project Data:   
ICR Review Date Posted:
09/18/2013   
Country:
India
PROJ ID:
P075060
Appraisal
Actual
Project Name:
India: Reproductive & Child Health Second Phase
Project Costs(US $M)
 2,222  4,475
L/C Number:
C4227
Loan/Credit (US $M)
 360  370
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
 452 338 20  260 25 25
Cofinanciers:
DFID, Foreign sources (unidentified), UN Fund for Population Activities
Board Approval Date
  08/22/2006
 
 
Closing Date
09/30/2010 03/31/2012
Sector(s):
Health (97%), Other social services (1%), Sub-national government administration (1%), Central government administration (1%)
Theme(s):
Population and reproductive health (29% - P) Child health (29% - P) Health system performance (14% - S) Other social development (14% - S) Decentralization (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Denise A. Vaillancourt
Judyth L. Twigg Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:


    Schedule 2 of the Development Credit Agreement states that
    The objective of the Project is to support the Borrower’s continuing Program for Reproductive and Child Health (the RCH II Program) that aims to achieve the reduction in maternal mortality and child mortality.

    The Project Appraisal Document/PAD (p. 5) states that
    The Project Development Objective (PDO) is to expand the use of essential reproductive and child health services of adequate quality with reduction of geographical disparities.

    Proxy indicators focused on the following essential services: contraceptive services; deliveries conducted by skilled personnel; child immunization; post-natal visits; and polio eradication. The PAD (p. 5) also notes that
    This proposed operation would help reduce maternal and child mortality and morbidity, lower fertility and the rate of population growth.

    This ICR Review will assess project performance against the higher-level objectives of the DCA (also mentioned in the PAD), as well as the service-level objectives and accompanying indicators specified in the PAD that contribute to those higher-level objectives.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

The RCH Project II was designed to support Government of India’s (GoI) National Reproductive and Child Health Program 2005/06-2009/10 (the RCH Program-Second Phase). The project’s three components (including their estimated and actual costs) are presented below.

1. Improvement in Essential RCH Services (Appraisal estimate: US$1,367 million; actual cost: US$3,081 million)
1.1 Activities Administered by the Ministry of Health and Family Welfare (MoHFW). This subcomponent was to support: (a) procurement of goods and services, particularly drugs, medical supplies and equipment; (b) routine immunization (including vaccines, cold chain equipment and maintenance, among other supports); (c) behavior change communication; (iv) training for improving technical and implementation skills; (v) expanding public private partnerships, with advice and support to states to this end; and (vi) policy development and pilots (private provider accreditation; social franchising, innovative financing schemes).
1.2 Innovative and Developmental Activities in State Program Implementation Plans (SPIPs). This subcomponent was to support innovative activities in states’ plans and provide flexible funds to support states’ creativity. Not defined ex-ante, this component aimed to support initiatives to expand access to family planning, safe motherhood services, newborn care, community-based child health and nutrition programs, promotion of adolescent health and development, urban RCH care, and service provision to special populations (tribal and scheduled caste).

2. Technical Assistance, Monitoring and Evaluation (Appraisal estimate: US$26.7 million; actual cost: US$30.0 million)
2.1 Technical Assistance (TA). Against the backdrop of a newly established MoHFW National Health Systems Resource Center (NSHRC) for the purposes of harmonizing TA contracting, this component aimed to support (a) states’ (especially Empowered Action Group/EAG and North-Eastern) planning capacity and (b) implementation of GoI’s comprehensive Governance and Accountability Action Plan (GAAP) to strengthen procurement capacity (both GoI and agent/UN agency). This component received parallel financing from DFID under a plan jointly agreed by GoI, DFID and IDA.
2.2 Monitoring and Evaluation. This subcomponent was designed to support a comprehensive M&E system to enable the close and regular monitoring of progress against Results Matrix indicators. Support would include two rounds of the Reproductive and Child Health Rapid Household Survey (2006-07 and 2009-10) to complement service statistics and community consultations. Special studies and polio eradication monitoring were also envisaged.

3. Polio Eradication (Appraisal estimate: US$829 million; actual: US$1,364 million)
This component aimed to support India’s efforts to achieve polio-free status, filling the program’s financing gap. A precursor to polio eradication, polio-free status was expected to be achieved by 2007, with subsequent supplemental surveillance activities envisaged for support for a few years after in order to declare polio eradication.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Costs. The actual project cost of US$4,475 million was double the appraisal estimate. Component 1 (essential RCH services) cost over twice the original estimate; Component 2 (TA and M&E) was 12 percent more expensive than planned; and the cost of Component 3 (polio eradication) exceeded the original estimate by two-thirds. The ICR does not explain the significant differences between planned and actual costs for Components 1 and 3, but they are likely to be associated with the two-year extension of the project (and program) (i.e., two additional years of operating costs).

Financing. The Bank, DfID and UNFPA pooled their funding and jointly disbursed against a broad subset of RCH II program expenditures, in accordance with pre-determined percentages through the RCH Project II. The IDA credit of 245 million SDRs was fully disbursed. DFID disbursed about 60 percent and UNFPA 125% of their respective planned financings (as expressed in US$ in Annex 1 of the ICR). Against the appraisal estimate of US$338 million in “unidentified foreign sources,” only US$25 million was disbursed (Annex 1 of the ICR), but the financier is not identified.

Borrower Contribution. GoI’s actual contribution of US$3,820 million was three and a half times the appraisal estimate of US$1,062.

Dates. The project was restructured in May 2010, including: a revision of the Results Framework and indicators (but not a change in PDOs); an extension of the project’s closing date by 18 months from September 30, 2010 to March 31, 2012; and a reallocation of the proceeds of the Credit to Category 2 for polio operating costs. Amendments to the DCA in 2008 (March and October) redefined categories to allow the use of Bank financing for operating expenses, TA for implementation of the Governance and Accountability Action Plan, and contractual staff and consultants. A fourth amendment to the DCA (March 2012) reallocated proceeds.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

High.The PDOs are highly relevant to India’s 12th Five-Year Plan (FY2013-17), whose overarching objective is faster, sustainable and more inclusive growth. The Plan advocates greater attention to inequality – both in outcomes and in opportunities – and ensuring access to good-quality health services. It sets specific targets for decreasing the rates of infant and maternal mortality (Country Partnership Strategy 2013). The PDOs are also highly relevant to the March 2013 World Bank Group’s Country Partnership Strategy, which is organized under three engagement areas: integration, transformation, and inclusion. Under “inclusion,” the Bank commits to strengthened public and private health delivery systems through the channeling of a larger share of health interventions to the state level, with an emphasis on disadvantaged states. The CPS focus is on strengthening institutions and accountability, developing local systems and capacities, private-public partnerships, and improved access and affordability for the poor and vulnerable. The PDOs (along with India’s Plan and the Bank’s Strategy) are well aligned with the country’s health issues: inequity of health status, with maternal and child mortality highest among the poorest and most vulnerable, persistent issues of service quality and inequitable access, and uneven capacity and performance across states.

b. Relevance of Design:

Substantial. The project’s operational objectives – to expand and improve key reproductive and child health services and make them more accessible and used, especially by poorest and most vulnerable groups, were directly supportive of the high-order objectives of reducing maternal and child mortality. The project’s support to key reproductive and child health services was appropriate to this end: contraceptive services, pre-natal care, deliveries by skilled personnel, post-natal care, and child immunization including efforts to eradicate polio. The project’s support to improving the availability, quality and utilization of these services – in terms of inputs and outputs – were also appropriate. They focused on both the supply-side (procurement of goods and services for the programs and services, such as vaccines, drugs, training, equipment, renovations and upgrading) and the demand-side (behavior change communications and financial incentives and other innovations to incite use). In addition, the design provided flexible funds to the states, with an emphasis on the most disadvantaged, to enhance their decision-making and resource allocation autonomy, in complement to state-level capacity building interventions. Project activities were appropriately oriented toward reducing geographic disparities by weighting funding in favor of the Empowered Action Group States, plus five additional states with poor maternal and child health status, and the north eastern states, which were generally worse off than other states as measured by socio-economic indicators. The design also appropriately embraced a program-wide approach, seeking to coordinate and consolidate external support around nationally articulated objectives and programs.


4. Achievement of Objectives (Efficacy) :


The program (and project) uses three main data sources: DLHS2 2002-04 for the baseline, DLHS3 2007-08 for the mid-point, and a UNICEF Coverage and Evaluation Survey (CES) in 2009 for an additional year of update. A DLHS4 was not conducted in time to provide end-of-project data.

Trends documented with these three sources are problematic for evaluating this project. First, they do not overlap sufficiently with the program implementation period. The program was launched in April 2005 and was extended to 2012, covering a seven-year period. DLHS 2 and DLHS3 document trends over the first half of the program period. The failure to complete the DLHS4 has resulted in the absence of trend data to measure progress during the second half of the program period.

Second, the comparability of the DLHS3 and the CES 2009 for assessing trends is uncertain. The Borrower, in its section of the ICR (p. 52), notes that the two sources are “not strictly comparable,” and this has been corroborated by the TTL, who notes that the sample sizes are completely different. Nevertheless, the ICR is pragmatic in drawing on this extra year of data and attempting to triangulate derived trends with its field visit findings.

The ICR is appropriately focused on program-level evaluation and the project’s contribution thereto, rather than on the project and attribution. Even though project implementation and disbursements were seriously delayed by one year or more after the project was declared effective in November 2006 due to a range of issues (ICR p. 8), the Bank can be credited with important contributions to the program from its very outset in 2005 on two fronts: (1) its substantial non-financial support during design and start-up of this program-wide approach, especially: technical and strategic dialogue with the Government; support to the consolidation of RCH II and the flagship National Rural Health Mission; the mobilization and convening of the support of other partners, both pooling and others; the active participation in Joint Review Missions; and extensive technical advice and support on governance issues; and (2) the project’s retroactive financing provision that allowed the reimbursement of operating expenses covering the early years of project implementation.

PDO: Expanded use of Essential Reproductive and Child Health Services, with reduction of geographic (and other) disparities: Modest achievement
(Despite apparently substantial achievement registered over the first half of the project, the absence of data/trends for the remainder of the project has caused a downgrade to modest.)

Outputs:
In the absence of end-program data, the ICR (p. 17) notes that after the mid-term review, project implementation geared up and made important contributions to the above-cited program goals, which would lead them to conclude that targets are likely to have been met or exceeded. Among the major contributions are:
In support of service expansion:
    • 25 states/union territories contracted the non-government sector to improve delivery of essential RCH services, surpassing the target of 15 states.
    • In 50 percent of districts in EAG states and Assam, the district hospitals conducted at least 20 C-Sections per quarter, falling short of the project target of 100 percent of districts in those states.
    • In 40 percent of districts at least one sub-district hospital conducted 10 C-Sections per quarter, falling short of the project target of 100 percent of districts.
    • 38 percent of 24-hour primary health care centers conducted more than 10 deliveries per month, not meeting the target of 60 percent.
    • Access to emergency obstetric care was enhanced through the establishment of first referral units (FRUs) and 24-hour services at primary health centers (PHCs). 8,475 PHCs were made operational 24x7 by March 2012, as against 1,263 in March 2005; and 2,315 FRUs were functional in March 2012, as against 955 in March 2005. The facilities providing these services were upgraded, but there remain questions about their quality (ICR p. 9).
    • 32,291 nursing and midwifery personnel were trained in skilled birth attendance as of September 2011. 1,070 medical officers were trained in life-saving anesthesia skills, and 601 medical officers were trained in comprehensive emergency obstetric care, including C-Sections. Ten-day trainings on basic emergency obstetric care have been initiated in the states. Master trainers are available in most states.
In support of reaching vulnerable groups and disadvantaged states:
    • 100 percent of state plans contained specific activities to reach vulnerable groups, surpassing the target of 75 percent (2009).
    • The number of rural health facilities and providers was increased. Additional auxiliary nurse midwives and nurses were trained, more than 600,000 community health workers (ASHAs) were recruited, and facilities were expanded to offer emergency obstetric care and blood bank/storage units.
    • Improved access to care for vulnerable groups was supported through the work of community health workers’ outreach and a cash transfer scheme to encourage women to deliver in health facilities.
    • Mobile health clinics in trial blocks were found to have increased access to RCH services. However, the ICR also notes (p. 25) a lack of effective trainers for enhancing sensitivity to gender and equity issues and significant variation across states in prioritization of services for vulnerable groups.
In support of demand stimulation:
Two synergistic interventions were found to be effective in imparting the positive trend in the use of assisted deliveries/institutional deliveries.
    • Under RCH II a conditional cash transfer scheme provided poor, vulnerable pregnant women with $35 to cover the expenses associated with their care during and following their pregnancies, including the delivery.
    • Under the NRHM a newly established cadre of community health workers, comprised of young women from the villages, counseled families, encouraged the use of services, and improved access to the above-cited conditional cash transfers. These workers were rewarded with $4-5 under the conditional cash transfer for every referral.
Because of low state capacity, IEC activities planned under the program were only partially implemented (mostly mass media campaigns, with behavior change communications relatively neglected). IEC was thus assessed not to be an important contributing factor to improved use of services.
In support of quality improvements:
    • Quality Assurance Committees were constituted and notified at the state and district levels in all states. Many states have developed checklists for monitoring. Several individual states established initiatives to improve the quality of RCH services: Gujarat has a quality assurance manual for RCH with a focus on total quality management and accreditation of PHCs and community health centers; and Madhya Pradesh and Odissa set up quality assurance programs, wherein district hospitals were strengthened to meet national accreditation standards. The National Health Systems Research Center successfully completed certification of 80 health facilities across the various states.
    • 55 percent of districts conducted training in the last three months of the project for skilled birth attendants, and 35 percent on integrated management of neonatal and childhood illness. This indicator was added during restructuring and neither baselines nor targets were specified.
    • The ICR reports that by closing all districts had at least one month’s supply of critical inputs (measles vaccine, oral contraceptive pills and gloves). Field visits indicate that essential drugs are now available in the system, but it is not clear if this indicator is regularly tracked.
    • Community monitoring, operational in 9 states by the time of the project’s closing, is noted by the Borrower to have revealed improvement in the quality of service delivery, infrastructure and service utilization, though no data are provided.
    • However, both DfID and UNFPA highlight quality improvement as an unfinished agenda (ICR, pp. 67-68).
In support of state-level capacity building:
    • 35 states/Union Territories successfully completed the institutional mobilization phase by 2010, meeting the target.
    • All EAG and NE states were visited by the MoHFW state facilitation teams, meeting the target.
    • Significant fiduciary capacity was built.
In support of central-level capacity building:
    • Both the program-wide approach and the Detailed Implementation Review (DIR: a review of allegations of corruption in procurement under RCH I; see Sections 5 and 11.b) have culminated in improved capacity for program management, more proactive monitoring and support of high-priority states, and improved capacity in procurement and financial management.
In support of polio eradication efforts:
    • Stool samples were collected from at least 90 percent of acute flaccid paralysis cases within 14 days, surpassing the target of 80 percent.
    • The target of non-polio acute flaccid paralysis rate of at least one case per 100,000 children below 15 years of age (an indicator of surveillance sensitivity) was reached, with a rate of 6.32 per 100,000 children.

Outcomes
    • Early project (2002-04 to 2007-08) trends show improving trends in the use of any contraceptive method, during the first half of the project, increasing from 45 percent to 47 percent, against an end-of-project target of 52 percent.
      • Modest increases for permanent methods (from 34 percent to 35 percent against a target of 36 percent) and for spacing methods (from 11 percent to 12 percent against a target of 16 percent) are reported in the ICR p. 18). It is not clear whether these very small changes of one percent are statistically significant.
      • Rates for scheduled caste groups increased from 43 percent to 49 percent, exceeding the 45 percent target, while rates for scheduled tribal groups increased from 39 percent to 42 percent, approaching the target of 45 percent.
      • EAG States were slated to increase from a baseline of 33 percent to 40 percent. By the mid-term four of the eight states had achieved the target, but the ICR does not present the baseline for each of the eight states.
    • Expanded use of delivery services, with reduction of disparities, exceeded targets by the project’s midpoint. Available data show that trends were improving between 2002-04 and 2007-08, and continued to improve, as documented by the CES (2009) and corroborated by field visits in two states (MP and Karnataka) and by reviews of Joint Review Mission reports.
      • Expanded use of delivery services conducted by skilled providers exceeded targets. Overall increases were from a level of 48 percent in 2002-04 to a level of 53 percent in 2007-08, rising to 76 percent in 2009, against the target of 60 percent.
      • Reduction of disparities in the use of skilled delivery services also exceeded targets. The 2002-04 level for EAG states was 32 percent. Levels in 2007-08 had risen to a range of 25 to 53 percent, and these further increased to a range of 47 to 79 percent in 2009, indicating more important increases than for the general population. The overall level of 66 percent for EAG states achieved in 2009 exceeded the program target of 45 percent. Gains were also impressive for the lowest wealth quintile, whose use of skilled delivery services rose from 28 percent in 2002-04 to 48 percent in 2009 (no specific target had been set for this group and no 2007-08 data were provided).
      • From an overall baseline of 35 percent, actual use of skilled delivery services rose in 2009 to 76 percent for Scheduled Castes and 61 percent for Scheduled Tribes, exceeding the target of 40 percent.
      • Proportion of institutional deliveries also increased overall (from 40% in 2002-04 to 47 percent in 2007-08 to 73 percent in 2009), with most EAG states having increases greater than those for the overall population thus revealing important headway in reducing disparities.
    • Expanded use of pre- and post-natal services, with reduction in disparities, was substantially achieved.
      • The proportion of women with at least three antenatal visits increased overall (from 50% in 2002-04 to 50% in 2007-08, rising to 63 percent in 2009), with most EAG states having increases greater than the increase for the overall population, thus revealing important headway in reducing disparities.
      • The percent of mothers and newborns visited within two weeks of delivery by a trained community-level health provider is reported to have increased (from less than 10 percent in 2002-04 to 50 percent in 2007-08, exceeding the target of 40 percent).
    • The percent of children 12-23 months of age who were fully immunized was on the rise in the early years of program implementation.
      • Between 2002-04 and 2007-08 DLHS data show that:
        • Immunization of females increased from 44 percent to 60 percent (project target: 75 percent), and
        • Immunization of males increased from 45 percent to 62 percent (project target: 75 percent).
      • CES data reveal progress in equity of results between 2002-04 and 2009:
        • Immunization of schedule caste groups rose from 42 to 59 percent (project target: 75 percent).
        • Immunization of schedule tribal groups rose from 37 percent to 50 percent (project target: 75 percent).
        • EAG states actually reached the project target of 60 percent in 2009, rising from 28 percent to 61 percent. Only two of the eight EAG states achieved the target by 2007-08 (DLHS3), and by 2009 all EAG states had achieved the target.
        • Immunization rates among the lowest wealth quintile also increased from 31 percent to 47 percent (no project target set).
    • Polio targets were exceeded under the program (validated data from National Polio Surveillance Project).
      • By 2012 at least 98 percent of households with eligible children were covered during national and subnational polio immunization days in high-risk districts. This surpassed the target of 80 percent.
      • India has been certified by WHO as having achieved polio-free status as of January 2012.

Impact: Unevaluable
Maternal, infant and child mortality and total fertility are affected by a broad range of socio-economic factors, improved reproductive and child services being only one among many others. Given the timeframe of program implementation and the unavailability of health services and impact data after 2010, it is difficult to assess with confidence the extent of the program’s contribution to documented trends in these outcomes, as well as the trends in these outcomes themselves for the project's latter years. Nevertheless, because of the good design of the program, it is plausible to assume that the program might be contributing to these outcomes, especially in the later years when implementation was more accelerated, and in the post-project years. This can only be adequately assessed once trends in services and outcomes are established for the later years of the project (and post-project). Available trends are as follows (National Sample Registration Surveys – SRS):
    • Maternal mortality declined from 301 in 2001-03 to 254 in 2004-06, further declining to 212 in 2007-09.
    • Total fertility declined from 2.6 to 2.5 between 2008-10, with most states showing a decline of 0.1 and 0.2 points.
    • There was an overall decline in infant mortality from 58 in 2005 to 47 in 2010, with wide variance across states.
    • Under-five mortality has changed very little between 2008 (60) and 2010 (59), although some disadvantaged states with higher than national averages have shown significant rates of decline.

5. Efficiency:

Substantial.
The project design encouraged allocative efficiency – of both Government budget and external pooled financing – on a number of fronts. First, it supported and prioritized high impact, cost-effective reproductive health services that were major components of national programs. Second, it emphasized the channeling of support to low performing, disadvantaged states with the highest rates of maternal, infant and child mortality, where the potential for impact was greatest for boosting India’s average rates. Third, its program-wide approach encouraged a coordination and consolidation of donor support through pooling of significant funding and coordination across a range of major donors. Fourth, the flexible funding to the states allowed them the opportunity to allocate resources to respond to the local challenges and context. Fifth, it enabled the mobilization of adequate support for polio eradication efforts and the allocation of a considerable portion of these funds to the two states where polio cases were still being reported (Bihar and Uttar Pradesh).

The program-wide design also encouraged the coordination/consolidation of big-picture activities and events, such as annual program reviews and joint missions at the central level, quarterly reviews and joint support to the states, and the coordination of massive amounts of technical assistance available for the program.

Inefficiencies in program implementation were encountered. There was a confusion of roles, responsibilities and procedures between the Government’s long-established RCH II program and its newly established, broader umbrella program National Rural Health Mission (NRHM), within which RCH II was an important component. Project support (both technical and financial) helped build and nurture synergies between the two programs, including the mobilization of significant additional funding, major policy and programmatic shifts, and enhanced supply- and demand-side interventions (ICR, p. 24). State and district capacity were initially overwhelmed with the management requirements of these two programs, which are gradually being consolidated.

Additionally, there were important setbacks in project implementation caused by the Detailed Implementation Review (DIR), which raised issues of potential corruption and required the negotiation of a detailed action plan to improve the rigors of fiduciary requirements and practice. There were weaknesses in procurement and financial management at both the central and state levels, including staff capacity and turnover. These short- to medium-term inefficiencies were a consequence of deliberate project design and post-DIR strategy, which chose to invest in the nurturing and use of national capacity systems and capacity; and they were anticipated in the risk assessment. The ICR (p. 7) provides evidence that through these DIR-related struggles and setbacks, system-wide capacity has been and continues to be built through a learning-by-doing process, and has culminated in improved use of project resources. Procurement standards are put into place to ensure the best value for money in the acquisition of goods and services, as well as transparency in the whole procurement process. The DIR uncovered irregularities in procurement under RCH I, which were undermining the efficient and transparent use of resources and raised concerns of potential corruption. They were addressed through capacity building at central, state and district levels, a decision to stop financing state-level procurements until state capacity was raised to a minimum standard and states would no longer resist post-reviews, and other health sector measures agreed with GoI to remedy and/or mitigate risks to fraud and corruption. While the DIR did slow down project implementation, the counterfactual of no DIR would have made the use of project resources much less efficient.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


The relevance of the PDO is high and the relevance of project design is substantial. The efficacy of the objective to expand and increase the utilization of key RCH services appears to be substantial, but downgraded to modest in the absence of end-of-project data. The efficacy of the objective to achieve polio-free status is high. The efficacy of the achievement of declines in MMR, IMR and U5M and the program’s contribution thereto is difficult to assess in the absence of end-of-project data and trends, both for services and impact levels. Had end-of-project data been available, it is likely that efficacy would have been substantial. Efficiency is substantial overall.

IEG typically does not rate outcome in the satisfactory range when efficacy is rated as modest. But, in this case, all documented service indicator trends were moving in the right direction during the first half of the project period, and accelerated project implementation after the project's mid-term and related project outputs provide some indication that project objectives are likely to be/have been achieved. (Harmonized guidelines define outcome as the extent to which PDOs "...were achieved or are expected to be achieved, efficiently.") The modest efficacy rating is linked to the lack of substantiated evidence of program efficacy, notwithstanding the project's positive trends during the first half of its life, and prospects of continued good trends during the latter part of the project. In this special case, a moderately satisfactory (vs. moderately unsatisfactory) outcome rating is warranted.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The activities supported by RCH-II project were continued by the government's National Rural Health Mission program. As such, it is expected that the GoI and partners will continue a strong commitment toward the health and RCH agenda through this program. The Borrower's funding represented half of the cost of the project, and it is likely that sustained improvements in health outcomes (both maternal and child health outcomes) observed over the last half decade will continue with or without World Bank support. Also, the program's focus on results and technical rigor contributed to building capacities at central and decentralized levels for sustaining program momentum and results in terms of fiduciary management and procurement. Programmatic and M&E capacities at state levels have improved, but are still in need of further strengthening. The ICR (p. 28) does note that the recent slowdown in the economy in India may strain national budgets, and allocations for social sectors like health and education may be at risk. Demand-side interventions (especially conditional cash transfers for stimulating use of delivery services and community health workers' outreach) are likely to have increased ownership by the population.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:

The project objectives and design were – and remain – consistent with national priorities and with the Bank’s CAS. The project was grounded in solid epidemiological evidence and lessons from RCH I. RCH II’s Program Implementation Plan was developed through an extensive consultative process led by MoHFW with active participation from states, NGOs, civil society and all supporting DPs. Among the positive policy changes envisaged in this new program were: a greater degree of decentralization to states and districts and strengthened accountabilities for funds received from GoI; more explicit pro-poor focus; demand stimulation; public-private partnerships; improved programmatic convergence; and improved program implementation capacity at central, state and district levels. These areas were reflective of lessons learned under RCH I and incorporated into the project design. Project readiness for implementation was reflected in: an established process for appraisal of state plans; agreement on GoI’s Governance and Accountability Action Plan; agreed procurement structure, system and plans for the first 18 months of implementation; agreed and established financial management arrangements; prepared and disclosed plans for safeguard requirements; and a finalized Memorandum of Understanding between the GoI and pooling partners. Most of the recommendations of a QER were addressed in the design. A shortcoming of the project was the results framework, which did not capture all critical elements of the results chain (see Section 10a).

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

The most challenging task for the supervision team was re-establishing dialogue with the GoI and DPs, focusing on the technical and programmatic issues, while also keeping due attention on the governance and accountability agenda. The team was adept and proactive in building a cohesive, constructive and functional DP group and in supporting program management and financial management post-DIR. Fiduciary actions were put in place to implement DIR recommendations and address weak procurement and implementation capacities at decentralized levels. Accelerated disbursement after fiduciary strengthening and the restructuring following the mid-term review are testament to the Bank’s supervision effectiveness during a delicate and volatile period in the wake of the DIR. Discussion with the project team provided additional evidence of the Bank’s stamina, proactivity, and political savvy in insisting on fiduciary rigor, all the while slowly restoring its technical dialogue during a very tense period. The Bank (along with other partners) was persistent in encouraging GoI to undertake the end-of-project survey, but was less effective, with less leverage, than was the case for fiduciary exigencies.

Supervision quality benefited from team continuity. Although the TTL changed after project approval, continuity and institutional memory were maintained with support from a senior co-TTL until 2009, who also mentored the new TTL.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government showed a strong commitment to reproductive and child health services. The launch of the Government’s National Rural Health Mission (NRHM) and its synchronization with RCH-II led to major policy and programmatic shifts, increased financial resources for the health sector, enhanced supply- and demand-side interventions, and monitoring and supervision mechanisms at state and district levels, all of which reinforced the implementation of RCH activities. GoI expenditures for health tripled from US$2.1 billion in 2005-06 to US$6.9 billion in 2012-13 (in nominal terms). Nearly 70 percent of this amount was allocated for RCH-related services. This financing allowed the central MoHFW to play a much more strategic role in guiding policy and instituting reforms than would have been otherwise feasible in decentralized India. This led to some confusion about the programmatic and fiduciary boundaries between the RCH II and NRHM, as evidenced by initial implementation delays at state and district levels. Weak fiduciary and program management capacities at decentralized levels limited the capacity to absorb these funds. Significant challenges to governance and accountability needed time to be addressed.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

MoHFW invested significant technical and management capacities during the design and implementation phases. As expected, building these capacities was a gradual process, especially at state and district levels. The first few rounds of the process of development of state-level project implementation plans were challenging. The quality of these plans was strengthened in subsequent years. However, building sustainable and high-quality implementation capacities at district and state levels still remains a challenge, especially in some of the EAG states. MoHFW did well in instituting Joint Review Missions, but continues to experience some ongoing challenges. The failure to conduct the DLHS surveys and release their results on a timely basis has seriously undermined program management capacity, and left the investors and implementers of this major program with no data to assess its outcome. States' performance as implementing agencies was mixed, but capacities were built through support and a learning-by-doing process. There were also issues with fiduciary compliance that were addressed and gradually being resolved (see Section 11b).

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The project aimed to support a comprehensive M&E system to monitor on a close and regular basis progress towards indicators laid out in the Results Matrix. This included repeat rounds of the Reproductive and Child Health Rapid Household Survey (in 2001-02, 2006-07, and 2009-10) and implementation of special studies and surveys, particularly for tracking polio eradication. Key elements of the results chain were not reflected in the M&E framework. For example, there were no baselines, targets or indicators for tracking the outcomes of behavior change interventions and other demand stimulation efforts. Innovations and pilots were an important part of the operation, but there were no targets or indicators to track the undertaking and exploitation of studies, the distillation and sharing of experience and lessons learned from pilots, or the taking of decisions to scale up successful pilots.

b. M&E Implementation:

There were important setbacks to the collection of data and the analysis of trends for tracking project performance and assessing efficacy. The Rapid Household Surveys were replaced by the DLHS surveys. The delay in the DLHS3 survey precipitated a delay in the mid-term review. The DLHS4 survey was also delayed and still not available. The consequent lack of end-project/-program data makes it impossible to establish and analyze trends over the life of the program/project – evidence against which efficacy could be credibly assessed.

Joint Review Missions were to have been undertaken every six months. Five actually took place by the time of the MTR, with efforts to consolidate recommendations for improved follow-up. Annual Joint Program Reviews were also undertaken, complemented by state-level reviews, funding of state plans being partially tied to their performance.

a. M&E Utilization:
Use of data for program management was mixed. Overall the richness of the DLHS data set was not fully exploited, with wide variation across states. Evidence indicates both use of M&E data and an increasing results focus at the central level. DLHS3 and UNICEF’s Coverage Evaluation Survey were used to identify and support 264 high-focus districts with poor health outcomes. Persistent M&E challenges include data quality and the use of data for management at state and district levels. Evidence from the ICR suggests, nevertheless, that a new culture of evidence-based programming is evolving at the central level, influenced by this project.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

    The project was classified under Environmental Category B under OP 4.01, with the following issues to be addressed: infection management in health facilities; treatment and disposal of medial waste; provision of water and sanitation facilities; and construction-related environmental issues. MoHFW undertook an Environmental Assessment and prepared an Infection Management and Environmental Plan (IMEP) in consultation with various stakeholders, the latter disclosed to the public. GoI also committed to conduct a capacity building program and establish a national-level working committee to guide MoHFW in plan implementation. Financial support for state-level implementation was to be provided through the SPIPs. The ICR does not assess plan implementation and related capacity building.

    In response to another safeguard trigger (Indigenous Peoples (OD 4.20, being revised as OP 4.10)) a Tribal Development Plan, nested in a Vulnerable Groups Health Plan (VGHP), disclosed to the public, outlines a broad range of interventions to increase access to reproductive and child health services in tribal areas, in coordination with private/NGO providers. In addition, traditional health providers would be trained and involved in the provision of RCH services. The ICR does not assess implementation of these plans.

b. Fiduciary Compliance:

Procurement arrangements under RCH II were designed to correct serious deficiencies in procurement processes encountered under RCH I (and addressed in depth through the DIR). An empowered procurement wing was established in MoHFW and a Governance and Accountability Action Plan (GAAP) was agreed. While there was reasonable progress in implementing the GAAP overall, there were issues and challenges related to procurement: lack of competition (case of RCH kits); unclear roles and responsibilities (between EPW and RCH division); issues with procurement procedures (causing IEC contracts not to be financed); lack of compliance with procurement procedures and states’ reluctance to allow post procurement reviews at the state level (resulting in DPs decision not to finance decentralized procurements); inadequate procurement and logistics/storage (with some equipment non-functional or not in use).

Financial management institutional strengthening efforts did contribute to progressive improvements. Among these are: creation, staffing and technical support of a Financial Management Group; contracting of a large pool of professionals and accountants at state, district and block level; an updated financial management manual and guidance; e-banking system for states and districts; a common IT-based accounting system; improved process for selecting external auditors. Performance and compliance vary greatly across states.

The post-DIR decision to limit pooling partners financing to operating costs significantly increased the transaction costs (for Bank and MoHFW) in determining eligible expenditures. After three years it was agreed to determine eligible expenditure based on acceptable audit reports from the states rather than on the six-monthly financial reports. This reduced administrative costs, but delayed the cycle of disbursement by a year. The improvement in the quality of auditing brought to light significant internal control issues in selected states/districts, leading to adjustments to disbursements in subsequent years. The ICR does not note whether audits were qualified.

In 2010 with sustained efforts from the Bank team, both the procurement and financial management functions improved considerably.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Negligible to Low
Negligible to Low
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

IEG drew on the evidence and project experience presented in the ICR to further distill the lessons presented in the ICR (pp. 31-33):
Investments in developing and using country procurement and financial management capacity and systems, and insistence on rigorous adherence to fiduciary exigencies during implementation, can damage or crowd out sector dialogue and depress the pace of project implementation. Yet this project shows that respectful persistence, coupled with technical support, can overcome these important setbacks and culminate in improved fiduciary capacity, a return to sector dialogue, and the promise of more sustainable country systems more capable of absorbing and effectively using pooled support.
Funding mechanisms can contribute to enhanced responsibility and accountability for the use of funds at the state and district levels as well as more flexibility and innovation at these levels to allocate and use these funds in a manner responsive to local context. Under this project, procurement and financial management capacity was gradually improved (and still is improving) through a learning-by-doing approach. This project also nurtured state- and district-level innovations that proved to be effective in supporting service-level objectives.
Delays in M&E implementation hinder the availability of reliable and timely sources of data to identify and implement mid-course corrections and evaluate overall program performance and outcomes. There appears to be little incentive for big programs to collect data and conduct evaluations. The Bank does not have the equivalent leverage (staff, guidelines and standards, pre-project conditions and legal clauses) for insisting upon rigorous M&E as it does for procurement and financial management, and thus its efforts to encourage GoI to undertake timely M&E were persistent, but less insistent. The Bank may want to reflect on how it can leverage more support and encouragement both to task teams and to Governments on this front.

14. Assessment Recommended?

Yes
Why?

The program's outputs and service-level outcomes, and their contributions to trends in maternal, child and infant mortality and fertility, can only be documented once end-of-project data become available. Such an evaluation may also demonstrate to GoI/MoHFA the value of timely M&E data for assessing and improving program performance and outcome.


15. Comments on Quality of ICR:


The ICR is well written and makes the best use of available data and information. It is also candid about the absence of end-of-project data and, in keeping with the guidelines, the need to “discount” the outcome rating in this light. Its assessment of a very complex program-wide approach is commendable, capturing well the challenges of this arrangement and the effect on project implementation and sector dialogue, especially in the aftermath of the DIR. However, the ICR does not state whether there was satisfactory compliance with safeguard policies, and whether audits were on time and unqualified.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions