Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Ar-provincial Maternal-child Health Investment Project (1st. Phase Apl)


  
1. Project Data:   
ICR Review Date Posted:
06/30/2011   
PROJ ID:
P071025
Appraisal
Actual
Project Name:
Ar-provincial Maternal-child Health Investment Project (1st. Phase Apl)
Project Costs(US $M)
 135.8  134.0
Country:
Argentina
Loan/Credit (US $M)
 135.8  134.0
Sector, Major Sect.:
: Central government administration, Sub-national government administration, Compulsory health finance, Health, Other social services
Cofinancing (US $M)
 0  0
Theme(s):
Health system performance (25% - P) Child health (25% - P) Population and reproductive health (24% - P) Social risk mitigation (13% - S) Indigenous peoples (13% - S)      
L/C Numbers:
L7225
Board Approval (FY)
  2004
Partners involved
 
Closing Date
12/31/2009 07/31/2010
         
Evaluator: Panel Reviewer: ICR Review Coordinator: Division:  
Pia Helene Schneider
George T. K. Pitman IEG ICR Review 1 IEGPS1

2. Project Objectives and Components:

a. Objectives:
This was the first phase of a three-phased APL. The overall objective of the ten-year APL program is identical with the objective of the APL1, with the exception of geographic targeting. The APL1 includes 9 provinces, whereas the full APL program is nationwide (PAD p.8)

The Legal Agreement states the development objective of the APL1 as “to contribute to the reduction of the infant and maternal mortality rate in the Borrower’s territory, as well as to introduce changes in the incentive framework of health care providers in the Borrower’s Participating Provinces, through the implementation of the MCHIP”.

The PAD (p.4 and 36) describes the main objective of the APL1 as “(i) to halt recent increases in the national rate of infant mortality (baseline at a national average of 16.8 per 1000 live births, and a NOA and NEA average of 25 per 1000) and then reduce it by at least 20% at the national level and at least 30% in the participating northern provinces over a period of 10 years, and (ii) change the dynamic of financing and providing health care services at the provincial level”.

The PAD objective and performance indicators do not mention maternal mortality reduction. Therefore, the ICR Review uses the DPO as stated in the PAD which is congruent with the results framework. The ICR used the same approach.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components (or Key Conditions in the case if Adjustment Loans):

I. Implementation of the Maternal-Child Health Insurance Plan Program (MCHIP) (appraisal estimate US$112.1 million, actual US$113.9 million). This component included:

  • Capitation payment based on number of MCHIP beneficiaries enrolled
  • Medical equipment for participating health facilities
  • Technical assistance and training for provincial Ministries of Health to manage MCHIP
  • Training of health care providers in delivering MCHIP benefit package
  • Information systems for providers and provinces to monitor implementation

II. Strengthening national and Provincial Ministries of Health Stewardship (appraisal estimate US$6.5 million, actual US$5.9 million). This component included:
  • Reorganizing the MSN‘s and PHM‘s mandates, staffing and relationships
  • Improving epidemiological information, financial and human resource management systems
  • Streamlining the regulatory and planning capacity of MSN and its dependent public health program units
  • Conducting studies essential for MSN policy formulation
  • Preparing Phase 2 of APL.

III. Communication and Community Outreach (appraisal estimate US$4.2 million, actual US$2.9 million). This component included:
  • Dissemination of detailed information about program among major stakeholders
  • Community outreach to increase participation of target population (e.g. information campaigns adapted to language and cultural needs of indigenous population)

IV. Program Monitoring, Evaluation and Auditing Systems (appraisal estimate US$3.9 million, actual US$6.6 million). This component included:
  • Information Technology (IT) to upgrade the monitoring of health provider performance, aggregation and reporting of information
  • Concurrent auditing of provider and provincial information for results based payment
  • Evaluation activities (e.g. baseline, mid-term and end-of-project impact evaluation)

V. Project Management and Administration (appraisal estimate US$1.4 million, actual US$1.9 million). This component financed the National project implementation unit (PMU) and the nine Provincial PMUs.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Costs:

  • The APL in the total IBRD loan amount of US$435 million is to be implemented over ten years (PAD p.5). The PAD indicates an IBRD amount of US$135.8 million for APL1; followed by US$149.2 for APL2; and US$150 million for APL3.
  • The APL1 disbursed 98.6 percent of the appraised amount. Per request of the Government, US$1.57 million were cancelled due to misprocurement.
Borrower:
  • The borrower co-financed the program through the federal and provincial government budgets in the estimated amount of at least US$154.1 million (PAD).
Financing:
  • Originally the 9 provinces agreed to co-finance 60% of the capitation amount to MCHIP. This share was reduced to 30% and delayed. Another project (H1N1) financed medical equipment originally planned under this APL1.
Dates:
  • First Loan Amendment on July 7, 2008: to (i) within Component I re-allocate funds from medical equipment to capitation payments, and expand MCHIP benefit package, and (ii) maintain 100% national co-financing of capitation payment until Dec 2008 and postpone start of 60% provincial co-financing to January 2009.
  • Second Loan Amendment on Oct 27, 2009: The APL1 closing date was extended by 7 months from the original closing date of Dec 31, 2009 to July 31, 2010. The reason for the extension was initial delays in MCHIP enrollment by beneficiaries, based on which the APL1 funds were disbursed to the Government budget. Provincial co-financing was reduced from 60% to 30% of the capitation amount.


3. Relevance of Objectives & Design:


Objectives - Substantial. Reducing infant mortality and changing the dynamic of financing and providing health care services at the provincial level are relevant in the nine Northern Provinces, given their relatively higher poverty and MDG mortality rates, and resource constraints for basic health care services. While several factors influence infant mortality rates in addition to basic medical care, adding this indicator focuses the attention of policy makers (and subsequent public finances) to MDGs; however, it is difficult to establish any direct impact of the APL1 on mortality as many other factors affect infant mortality rates. The CAS (2004) overall objective was to help reduce the extent and severity of poverty in Argentina after the crisis. The APL development objectives were relevant to the CAS both at appraisal and currently (CPS 2010-2012). The latter stresses a continued focus on strengthening provincial health systems through the expansion of the MCHIP, and concurrent audits to support the system's fiduciary performance and transparency. Changing the health financing mechanisms with the introduction of MCHIP remains highly relevant as it provides essential information to the federal government and provinces on the use of health funds that is used to set financial incentives to providers and provinces to improve care in line with standard treatment protocols.

Relevance of Design - High. The design was highly relevant and innovative to achieve the DO. A ten-year APL program helps prioritizing the MCHIP first to the poorest regions while preparing the remaining provinces for the scale up. The design incorporates lessons learned from past projects and an ESW on health financing reforms. The four technical components embrace a system-approach and cover activities (e.g. patient-level data collection and audits) often neglected in projects. The design included innovative parts with conditioned disbursement to the Government budget based on the number of beneficiaries enrolled in MCHIP, "tracer" results-linked capitation transfers to provinces and substantial investment in patient-level data collection and evaluation to link payments to service delivery. The results framework reported relevant key indicators to assess progress. In addition, the 9 provinces measure monthly the number of beneficiaries enrolled in MCHIP for the capitation payment and quarterly the 10 "tracer" health service indicators to assess treatment compliance with standardized protocols. Results are used to adjust the fiscal transfer to provinces thereby setting a financial incentive to policy makers to enable providers in better service provision. The design thus managed to institutionalize the M&E results framework into the country's fiscal transfer system thereby strengthening overall system governance.

4. Achievement of Objectives (Efficacy) :

1. To halt recent increases in the national rate of infant mortality (IMR) and then reduce it by at least 20% at the national level and at least 30% in the participating northern provinces over 10 years - Modest

Note the PAD was written in 2004 and referred to 2003 baselines, however, the APL1 became effective in Nov 2004, thus 2005 will be taken as baseline for comparison in this ICR-Review. By June 2005, the MCHIP had about 30% of the target population enrolled, and compliance with tracer indicators were still very low (e.g. well-child visits (10%), APGAR score (3.7%) etc.). The IMR decrease before 2005 cannot be attributed to the APL1.

  • Infant mortality rate (IMR) decreased nationwide from 13.5 in 2005 to 12.1 in 2009 (9% reduction in 4 years could possibly reach target of 20% reduction over 10 years).
  • IMR in NOA and NEA decreased from 16.7 in 2005 to 14.6 in 2009 (12.6% reduction in 4 years is not on track with target of 30% reduction over 10 years).

While IMR is an important objective, attributing the impact of MCHIP on IMR is ambiguous. IMR has shown a decreasing trend for at least 20 years nationwide and in comparator countries. Also, average IMR are based on data from MCHIP beneficiaries and non-members, suggesting that various factors have affected the average decrease, not just MCHIP.

Two tracer indicators that are influential for IMR have yielded the following results:
  • Early inclusion of pregnant women (<20 week) in prenatal care program increased from an average of 4% in 2005 to 45.4% in April 2010 in all nine provinces (tracer 1)
  • Well-child visit rate for children up to 12 months increased from 9.5% in 2007 to 31.5% in April 2010.


2. To change the dynamic of financing and providing health care services at the provincial level - High
  • All 9 provinces implemented the financing change through capitation financed MCHIP and tracer payment based on 10 tracer results.
  • Audits of tracer results and provider reports were conducted as planned and results used to make corrections.
  • MCHIP enrollment increased from 0% in 2004 to an average of 84% in 2010 in all 9 provinces. All provinces surpassed target of 50% enrollment.
  • The share of providers contracted by MCHIP increased from 0% in 2004 to 71% in 2010 (surpassed target of 50%).
  • The percentage of tracer targets met by the nine provinces increased from 0% in 2004 to an average of 74% in 2010, ranging from 54% in Catamarca to 90.5% in Tucuman (surpassed target of 50% for each province).

5. Efficiency (not applicable to DPLs):


Overall Efficiency - Substantial

Project funds were implemented efficiently on time with only a six-month project extension. Project spending focused on primary and preventive care services provided in basic health facilities, helping to avert higher cost hospital admissions. The APL1 funded several efficiency-enhancing activities, including better patient level data to improve the adherence to standard treatment protocols.

At appraisal, an economic analysis was conducted, estimating the project’s net present value at US$ 670 million with a rate of return of about 18.1% over ten years.

The ICR estimated a substantially higher rate of return of 31.6% and a benefits/cost ratio of about 2 over 6 years. This result is not plausible considering that the ICR estimated is based on a lower reduction in the infant mortality rate than the reduction estimated at appraisal.


a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:



Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
18.1%
100%
ICR estimate:
Yes
31.6%
100%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The APL1 was strong in setting up and institutionalizing a new organizational and financing structure and institutional framework under the MCHIP. Based on the sub-ratings of substantial relevance of objectives, high relevance of design, modest and high outcomes, and substantial efficiency, IEG’s overall rating of project development outcome is satisfactory. The project had a clear poverty and gender focus and targeted indigenous groups.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating :

The risk to the DOs of reducing infant mortality and changing the health financing and provision of care is rated as low. Infant mortality continues to decrease as a result of improved availability of basic health care to the poor and more sophisticated services (e.g. cardiopathy treatment etc), reduced poverty and other factors such as increased female literacy rates and higher employment rates.

The Government shows strong ownership for the MCHIP and related reforms in health financing and provision of care. The MCHIP is well received by the population and supported by the current central and provincial political leadership. It has been expanded nation-wide based on the experience from the nine provinces. The remaining 14 provinces supported under the APL2 had 64 percent of the target population enrolled in 2010. The country's 24 provinces have already signed an agreement that will lead to the expansion of the MCHIP to become a Provincial Health Insurance.

However, provincial governments differ in their political and financial support to MCHIP; this may affect the future sustainability of the program on a provincial level. This question is currently addressed under the APL2.


a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

Ensuring Quality-at-Entry: The project was well-prepared. The Bank team included several well-experienced staff who provided high-level technical advise and drew lessons from experience. The collaboration with the Government was highly satisfactory. The APL program was designed to strengthen the implementation of specific aspects of the Government health reform program (HSRP) supported under the Bank Health Sector Adjustment Loan (SECAL) and thus strategically well-anchored. There is strong government ownership for the MCHIP. The APL1 focused on the needs of women and children in lower-income groups and indigenous groups during the economic and fiscal crisis. The Loan was innovative as it was the first time that a Bank health sector operation added a fiscal transfer for health to include a results-based component targeted to improve health service delivery for poor households. The design was comprehensive; it included health financing reforms, had an extremely strong component for data collection, monitoring and evaluation and auditing, and included investment in the provision of care that were crucial for providers and the payment reform to be sustainable.

Quality of Supervision: The Bank team provided high-quality supervision under the APL1 with a technically strong and professionally experienced team. The team maximized the impact of the APL1 by complementing parallel investment operations and preparing the scale-up to APL2. The Bank team took actions when design adjustments were needed and set up a comprehensive M&E framework to evaluate progress made. This has influenced the institutionalization of M&E and auditing systems at the national and provincial level. Later, the team secured additional Trust Funds to evaluate the impact of the MCHIP. The team provided excellent technical follow-up on the reforms in health financing and provision of care which were rather ambitious and sophisticated even in a middle-income context. This follow-up was key for the success of the project. When misprocurement was identified during the procurement supervision the Bank team took the correct steps, involved INT per Government request, intensified supervision, and developed and implemented a Procurement Action Plan, and cancelled the related project amount. Client relationship in all 9 provinces and the central MOH were highly satisfactory.

a. Ensuring Quality-at-Entry: Highly Satisfactory

b. Quality of Supervision: Highly Satisfactory

c. Overall Bank Performance: Highly Satisfactory

9. Assessment of Borrower Performance:

Government Performance: The Government has implemented the APL1 at the national and provincial level. Despite changes in leadership, the central MOH continued to show high commitment, ownership and strong leadership for implementing the MCHIP in the 9 provinces and scale-up in other provinces. Initially low enrollment rates were addressed by the Government by making MCHIP membership compulsory for child benefit recipients since January 2009. The increased co-financing level by the Provinces has been slower than originally planned and a stronger financial commitment by the Provinces will be needed for the Plan to be sustainable. The MOH took the right steps and immediately alerted the Bank to request an INT investigation when misprocurement was identified during supervision. The Government worked actively with the Bank on a governance and anti-corruption action plan to mitigate fiduciary risks following the INT investigation.

Implementing Agency Performance: The APL1 was implemented by a national PMU, 9 provincial PMUs and the UFI-S, a unit at the MOH responsible for fiduciary aspects. The PMUs at the national and provincial levels are technically strong and well trained. The national PMU provided substantial technical assistance to both provincial MCHIPs and providers, especially in provinces with weaker institutional capacity. The national PMU benefited from continuity in technical and management leadership which ensured strong collaboration with all stakeholders. The national PMU was also strong in M&E aspects which was key for the results-based financial transfer to the provinces.

The ICR notes (p.34) that the national PMU underperformed in implementing the communication strategy. The procurement function in the MOH procurement unit UFI-S was temporarily weak when misprocurement was identified (see procurement section) which led to the cancellation of US$1.57 million of project funds. However, the Government immediately requested an investigation by the Bank when it suspected issues of misprocurement, and implemented corrective measures in line with Bank requirements.

a. Government Performance: Satisfactory

b. Implementing Agency Performance: Moderately Satisfactory

c. Overall Borrower Performance: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

Design: The M&E design was innovative in combining data needs for project M&E with the needs for the health system. The project M&E design was to the point with explicit objectives, input and output factors and expected results to assess progress and outcomes. The DOs include one health outcome indicator - reducing infant mortality rate - which is technically not an appropriate indicator; but can be justified based on political grounds to get the attention of policy makers to MDGs and negotiate related budget increases. The project supported the implementation of this design with substantial investment in data systems, collection and analysis, and an explicit project component for concurrent audits by an independent firm. Results were used to make adjustments to the project design (e.g. change the trazadores payment). The design included an impact evaluation.

Implementation: The financing change requires substantial investment in patient-level data collection and analysis. The project financed IT and software investment in health facilities, province and central administration. The project invested substantially in the collection and management of valid and reliable patient level data in health facilities. The project M&E was implemented, the impact evaluation conducted, and independent concurrent audits take place to verify validity of data. An independent private auditor firm has been contracted under the project by the federal PMU to conduct concurrent audits of enrollment data and compliance with indicator results in provinces.

Utilization: Data on MCHIP membership are analyzed to calculate the capitation payment, and patient-level data are analyzed to define the payment to provinces based on ten health indicators and the fee-for-service payment. Provincial MCHIP identify for each provider all patients who are not in compliance with treatment. This information is passed regularly to providers who will follow-up with non-compliant patients through their outreach programs. It helps providers to adhere to standard treatment protocols and increases compliance with standard treatment protocols. Together with the independent audit system, this provider performance analysis is a particularly strong and unique feature of the health project.

According to the CPS 2010-2012, the concurrent audits designed to ensure fiduciary standards of the Bank-financed results-oriented MCHIP has gradually become the auditing standard for the health insurance scheme at the national level and is envisaged to be adopted in future Government programs in the sector.

The impact evaluation is not completed yet, and the authors indicate that data and methodological constraints do not allow policy recomments based on the evaluation; thus, it's policy relevance is unclear. A household survey will be launched in August 2011 to examine the impact of the MCHIP based on reliable data.

a. M&E Quality Rating: High

11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts):


Environmental Safeguards: The project was rated as Category C as it did not involve new construction on new sites. Nevertheless, an environmental assessment was conducted to strengthen the capacity of the MOH in this area.

Social Safeguards: The Indigenous People Safeguard Policy (OD 4.20) was triggered given the large share of indigenous people living in NOA and NEA. An Indigenous People Implementation Plan was developed by the Government and adhered to. One of the 10 tracer indicators was specifically targeted to assess inclusion of indigenous patients.

Fiduciary:
- Financial Management was rated Satisfactory throughout project and all audited financial statements were acceptable. A Bank Review conducted in 2008 recommended specific actions to strengthen FM which were implemented.

- Procurement was rated Satisfactory until June 2008. Then it was downgraded to MU when a procurement mission and a subsequent investigation by INT requested by the Minister, identified misprocurement in 3 cases which led to cancellation of project funds (see above). A procurement action plan was developed and implemented to strengthen the MOH procurement department.


12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate


 

Bank Performance:
Satisfactory
Highly Satisfactory
Bank performance was exemplary during preparation and supervision and merits Highly Satisfactory. 
Borrower Performance:
Satisfactory
Moderately Satisfactory
A procurement mission and a subsequent investigation by INT requested by the Minister of Health, identified misprocurement in 3 cases which led to cancellation of project funds. The provinces were less forthcoming than originally planned in co-financing the MCHIP. 
Quality of ICR:
 
Satisfactory
 

13. Lessons:

  • Project M&E was exemplary, and the use of data was unique. This M&E model (including patient-level data and feedback of results to individual providers, independent audits through private firm etc) should be applied in all Bank projects with any kind of results-linked financing.
  • PDOs should not include health outcome indicators such as infant mortality rates given the attribution concerns, even if projects conduct impact evaluation.
  • The use of impact evaluation in policy making should be very clearly defined before funds are committed to conduct such analysis.


14. Assessment Recommended?

Yes
Why?
A comparative PPAR is conducted on Results-Based Financing in Argentina and Brazil. This APL1 shows some important design features and lessons based on which it has been selected for the PPAR.

15. Comments on Quality of ICR:

The ICR is well written, and by and large comprehensive. However, there are some methodology issues the ICR could have addressed more carefully (e.g using a baseline for comparing project DOs from 2 years before the project started, and being more critical about the data and methodology issues in the impact evaluation).

a. Quality of ICR Rating: Satisfactory

(ES-Rev5-Jul/06)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions