Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Hiv/aids Response Project


  
1. Project Data:   
ICR Review Date Posted:
10/16/2012   
Country:
Mozambique
PROJ ID:
P078053
Appraisal
Actual
Project Name:
Hiv/aids Response Project
Project Costs(US $M)
 64.0  N/A
L/C Number:
CH030
Loan/Credit (US $M)
 55.0  59.2
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  03/28/2003
 
 
Closing Date
12/31/2008 06/30/2011
Sector(s):
Health (40%), Other social services (35%), Central government administration (20%), Sub-national government administration (5%)
Theme(s):
HIV/AIDS (29% - P) Population and reproductive health (29% - P) Gender (14% - S) Participation and civic engagement (14% - S) Social safety nets (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Judyth L. Twigg
Robert Mark Lacey Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
According to the Development Grant Agreement (DGA, p. 17), the project’s objectives were “to assist the Recipient in the implementation of its National Strategic Plan through: (i) improving the institutional capacity of the National AIDS Council (CNCS), Participating Ministries, Civil Society Organizations, and the private sector to plan, deliver, and monitor HIV/AIDS response interventions; (ii) supporting Sub-projects aimed at HIV/AIDS prevention, care, and mitigation in local communities; (iii) supporting prevention and care programs for staff in the Participating Ministries and their client base at the regional and national levels; and (iv) strengthening and scaling up of health services for prevention of HIV infection and provision of care and treatment through integrated health networks.”

The Project Appraisal Document (PAD, p. 2) states the objectives in terms of assisting the Government with financing of its National Strategic Plan to Combat Sexually Transmitted Diseases (STDs) and HIV/AIDS. “This strategy is designed to slow the spread of HIV/AIDS infection and mitigate the effects of the epidemic, through prevention, care, treatment, and mitigation activities.”

The objective as stated in the DGA cites assistance with the implementation of the National Strategic Plan as the desired outcome, and then lists in (i) through (iv) a set of outputs intended to achieve that end. The DGA statement does not elaborate on the content of the National Strategic Plan. As the definition stated in the PAD more clearly states the key elements of the National Strategic Plan, this Review will assess achievement of the outcome-oriented objectives elaborated in the PAD: slowing the spread of HIV/AIDS infection, and mitigating the effects of the epidemic.

In June 2007, the project’s objectives were revised to indicate that the Bank was only one player in the fight against HIV/AIDS. The revised objective was “to contribute to slow the spread of HIV/AIDS in Mozambique and mitigate the effects of the epidemic through prevention and care activities” (Project Paper, p. 3).

Both the original and revised objectives point to the same two outcomes: slowing the spread of AIDS and mitigating the effects of the epidemic.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 06/27/2007

c. Components:
The project contained five components:

1. Community and Civil Society Initiatives (appraisal, US$ 28.0 million; actual, US$ 26.6 million). This component was to empower communities to respond effectively to the epidemic through mobilizing communities, promoting local initiatives, and strengthening the capacity of local actors. A Community and Civil Society Facility held by CNCS was to be created to finance appropriate HIV/AIDS-related activities carried out by eligible applicants, such as community and faith based organizations, non-governmental organizations (NGOs), the private sector, associations, and other organizations. The management of the Facility was to be decentralized to the provincial level.

2. Capacity Building for the Civil Society HIV/AIDS Response (appraisal, US$ 5.5 million; actual, US$ 3.8 million). This component would support efforts to create AIDS-competent communities by improving skills and increasing implementation capacity of implementers, supporters, and target groups active under the Community and Civil Society Initiatives. Activities to be funded included hiring and training “Facilitation Agents” in each province; courses and activities to build awareness and leadership capacity in the public sector, civil society, and private sector; and the formulation and distribution of training modules and packages of information, education, and communication (IEC) materials.

3. Government Multi-Sector Response (appraisal, US$ 7.0 million; actual, US$ 5.1 million). This component would support ministries and subordinate institutions in the development and implementation of HIV/AIDS-related programs directed toward their own personnel (particularly high-risk staff such as soldiers, police, extension agents, teachers, and health workers) and their families, as well as programs directed toward their clients. The program was to concentrate on non-health ministries and other key public sector institutions to respond effectively to the epidemic, with emphasis on prevention and care for infected and affected families. A Focal Point was to be appointed in each ministry. District Administrators were to foster development of District-level HIV/AIDS plans.

4. Strengthening and Scaling Up Health Sector Services for HIV/AIDS (appraisal, US$ 17.5 million; actual, US$ 19.5 million). This component was to scale up the health sector response to the epidemic and to provide technical leadership on treatment and care for people living with HIV and AIDS (PLWHA). It was to support: (i) strengthening of the Integrated Health Network providing voluntary counseling and testing (VCT) and HIV/AIDS-related services in four central provinces; (ii) increasing the supply of HIV-related drugs and supplies, condoms, drugs for opportunistic infections (OIs), anti-retrovirals for the prevention of parent-to-child transmission (PPTCT) of HIV, and post-exposure prophylaxis; (iii) strengthening clinical laboratory capacity for CD4 monitoring and diagnosis of OIs; (iv) measures to enhance bio-security; (v) measures to enhance blood safety; (vi) training and anti-retroviral drugs for post-exposure prophylaxis for health workers; (vii) improved management of OIs; and (viii) monitoring and evaluation of the component and the overall progress of the epidemic.

5. Institutional Development for Program Management (appraisal, US$ 6.0 million; actual, US$ 4.2 million). This component was to support the Executive Secretariat of the CNCS, strengthening its capacity to lead the country in the campaign against HIV/AIDS.


The components were not revised.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost:

US$ 31.47 million had been disbursed at the time of restructuring.

Actual total project costs are not specified in the ICR. As explained below, there was no actual Borrower contribution, but according to the project team, strong exchange rate gains more than compensated for the loss of that US$ 9 million, so that the project's final costs in US dollar terms were greater than the estimated costs, even without the US$ 9 million from the Government. However, the project team did not specify these final costs, and it is not known whether there were fees that should be added to the sum of the actual component costs (US$ 59.2 million) in order to calculate the total final costs.

Financing: The ICR (p. 23) states that the Bank’s total disbursements were US$ 59.2 million. However, on p. viii of the ICR it is indicated that US$ 60.39 million had been disbursed at the last Implementation Status Report (ISR) on June 29, 2011. The ICR does not explain the difference between the Bank’s planned and actual costs.

Borrower Contribution: There was a US$ 9.0 million planned contribution from the Government. The ICR (p. 23) states that the exact contribution is not known. The project team clarified that the Government contribution was intended to go toward taxes, but because of shortages of funds, the Government requested and was granted an amendment to the Grant Agreement changing the project to 100% Bank financing.

Dates:

On June 27, 2007 the project was restructured to modify the objectives, introduce a new financing modality in which Bank funds were pooled in a Common Fund with those of other development partners (the Canadian International Development Agency, the Department for International Development (UK), the Danish International Development Agency, the Global Fund, Irish Aid, and the Swedish International Development Cooperation Agency), amend some of the key indicators, and extend the closing date by one year to December 31, 2009.

In October 2009, the project was amended to modify its institutional and governance structure. A new Grant Management Agency (GMA) was established in the Ministry of Health to channel funds and facilitate implementation of project activities in participating ministries, civil society organizations, and sub-project beneficiaries; in effect, this GMA took over some of the roles and responsibilities of the CNCS. In addition, US$ 12 million was reallocated to MOH for scaling up HIV and syphilis testing, expanding condom distribution, strengthening links between HIV and reproductive health, and improving bio-safety and nutrition (explaining the greater-than-planned spending on Component 4). A further US$ 3 million was reallocated to the establishment of a Rapid Results Fund (RRF) for prevention; this RRF was supported by various donors and managed by the United Nations Development Programme, supporting proposals focusing on multiple concurrent sexual partnerships, reproductive health and HIV, HIV and tuberculosis (TB), and male circumcision. According to the ICR, this amendment also extended the project’s closing date by another year, to December 31, 2010.

The ICR does not specify when the closing date was extended for the final time, to June 30, 2011. The project team clarified that the second extension of the project's closing date was for eighteen months, not twelve, bringing the closing date from December 31, 2009 to June 30, 2011.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Relevance of the original and revised objectives is rated Substantial. The project’s objectives were substantially relevant to country conditions at the time of appraisal. It was estimated that Mozambique had one of the fastest-growing HIV epidemics in sub-Saharan Africa, with an adult HIV prevalence rate of roughly 12%. Multiple factors converged to create an environment of high vulnerability, including a long and protracted civil war, large population movements, proximity to high-prevalence countries that hosted Mozambique’s refugees, and a deterioration of physical infrastructure and social fabric. The objectives are also substantially relevant to Mozambique’s National HIV Strategic Plan (2005-2009, updated to 2010-2014), with pillars of prevention, treatment, care, coordination of the response, operational research, and monitoring and evaluation. Finally, the project’s objectives are substantially relevant to current Bank strategy. The project was part of the second phase of the Bank’s Multi-Country AIDS Program for the Africa Region (MAP), launched in 2000, whose objectives were to dramatically increase access to HIV/AIDS prevention, care, and treatment programs, with an emphasis on vulnerable groups. The Bank’s Country Partnership Strategy (CPS, 2007-2011), current at project closure, cites HIV/AIDS as “perhaps the single greatest threat to Mozambique’s continuing growth and development” (p. ii). One of the CPS’s three pillars is “Equitable Access to Key Services,” with increased access to information on HIV/AIDS and to treatment as the first listed outcome under that pillar.

b. Relevance of Design:
Relevance of Design under the original and revised objectives is rated Substantial. The PAD (pp. 45-50) contains a detailed results framework that plausibly links the project’s activities to its expected outcomes and achievement of development objectives. The PAD (p. 8) identified priority vulnerable groups, at highest risk in terms of both infection and impact: young people (especially girls), soldiers, miners, prisoners, drivers, and commercial sex workers. It also cited three main transport corridors as desirable geographic areas of focus. The PAD (p. 13) cites these groups as target populations, and the project design, particularly Component 3, specifically focuses interventions on these groups. It is not clear, however, that Component 1 contained incentives for community-based subprojects to focus on prevention interventions targeting high-risk groups. The 2009 restructuring, the creation of the RRF, and the allocation of resources toward male circumcision and multiple concurrent sexual partnerships made the design even more substantially relevant, as it incorporated the results of current research on HIV prevention into the project’s activities.


4. Achievement of Objectives (Efficacy) :

While the ICR recognizes the contributions of other donors to the fight against HIV/AIDS in Mozambique, particularly after the Bank entered the Common Fund arrangement in 2009, it does not provide information that would aid in an assessment of attribution of observed outcomes specifically to Bank-financed interventions. The Global Fund has made a series of grants to Mozambique: Round 2, 2006-2008, US$ 7.7 million for prevention and care; Round 2, 2005-2010, US$ 88.1 million for treatment and care; Round 6, 2007-2010, US$ 61.3 million for prevention, treatment, and care. The United States Government, through the President’s Emergency Plan for AIDS Relief (PEPFAR), provided US$ 835.6 million in comprehensive support for HIV/AIDS prevention, treatment, and care programs from 2004-2009. The project team clarified that the Bank's major contribution was as a convener, driving the multi-sectoral agenda and mobilizing actors across Government, as well as establishing the National AIDS Council and M&E structures around which all financing was provided. This review therefore assesses, in part, the overall efficacy of the program to which the Bank contributed.


Slow the spread of HIV/AIDS (Original and Revised Objective) is rated Modest. There is no evidence that the project reached high-risk groups most likely to spread infection. The project’s interventions related to male circumcision and multiple concurrent partnerships were well conceived, but took place too late in the project period to have likely contributed to observed outcomes during the project’s time frame. The project team added that prevention activities by the Ministry of the Interior targeting police, and by the Ministry of Education targeting youth, were strong, but specific data are not provided for these programs.
Outputs:

3,771 community-based subprojects were funded. The project provided capacity building for NGOs and civil society organizations, primarily through the hiring of Facilitating Agents. After the mid-2009 restructuring, two proposals were funded involving multiple concurrent partnerships, and two involving male circumcision, meeting the target. Overall, however, according to the ICR (p. 24), the community-based subprojects had limited focus on prevention, with the exception of “some targeting” of geographic areas with a high risk of HIV transmission (border areas and truck stops). The ICR does not provide information on what specific services were provided through the community-based subprojects, nor on their coverage. It does not discuss any prevention activities that were targeted specifically at high-risk groups most likely to spread HIV infection. The project team confirmed that most of the community-based subprojects were targeted at the general population.

According to the project team, after the mid-term review there was more discussion of the drivers of the epidemic, and therefore there was more interest in scaling up activities directed at high-risk groups: truck drivers, commercial sex workers, injection drug users, and men having sex with men. There was promotion of female condoms and treatment of sexually transmitted infections (clinics were opened in high-risk areas, aimed at truck drivers and sex workers, so that they could receive treatment when symptoms appear). The ICR provides no specific data for these efforts, however, and the project team confirmed that there is a need now to scale up these efforts.

557 private sector subprojects were funded, surpassing the target of 450, reaching an estimated 45,000 workers and family members. Most of these subprojects focused on IEC, peer education, distribution of condoms, and VCT.

Focal points were selected in 18 ministries and 20 public or para-statal agencies such as the National Institute of Health and the national airline. According to the ICR (p. 28), disbursements for activities were often late, and the focal points were part-timers who did not receive additional payment for their work on HIV/AIDS. The ICR states that this public sector component “did not consider the strategic role a ministry could play in the national response,” with activities relating only to AIDS in the workplace.

Multiple behavior change communication (BCC) sessions were organized through radio, television, and community theater in all provinces.

10,000 community leaders and 9,162 religious leaders were trained and/or reached through HIV/AIDS awareness interventions, against a target of 500 national leaders and 5,000 provincial leaders.

66.5 million condoms were distributed, exceeding the target of 30 million. The project team explained that condom distribution was primarily through health facilities, with coverage at health facilities almost universal, from the district to provincial level, and special attention to distribution to women of childbearing age. Distribution also took place at bars, kiosks, and markets, where condoms were sold at very low prices. Condoms were also made universally available at entrances to public institutions. Population Services International engaged in condom social marketing, through radio, TV, and other media.

359 voluntary counseling and testing centers were in place, exceeding the target of 100.

Blood banks received training in bio-safety. 100% of blood for transfusion was screened for HIV by 2008, meeting the target, but only 35.5% of blood units were tested under conditions of adequate quality control.

The project team added that the project engaged the Government in continuous policy dialogue on the prevention of transmission of HIV from parents to children (PPTCT), and financed training of staff, procurement of HIV tests, and other inputs to improve provision of PPTCT services.

100% of secondary and tertiary health facilities had the capacity to diagnose and treat sexually transmitted infections by 2007, reaching the target one year early. The ICR does not provide information on the actual use of that capacity to diagnose and treat these infections. The project team later provided the following data on numbers of cases of sexually transmitted infections notified by the Ministry of Health:
Syndrome20072008200920102011
Leucorrhea128,721125,363148,897152,975192,568
Genital Ulcer205,775190,126193,261148,035168,617
Urethral Discharge184,392173,489187,871109,689128,059

These data indicate overall downward trends in reporting of genital ulcers and urethral discharge, which could indicate effective use of installed capacity to treat infections. However, as the project team points out, these data should be interpreted with caution, as they represent only those individuals who sought care in health facilities, and a reduction in cases could represent a reduction in incidence or a reduction in utilization of services.

The project team also later added the following information regarding outreach efforts to high-risk groups (data drawn from the National AIDS Council's 2011 Joint Annual Evaluation Report). No information is provided on the coverage represented by these interventions.

  • In November 2007, the Government adopted a plan to accelerate prevention, prioritizing high-risk groups such as commercial sex workers, truckers, prisoners, members of the armed forces, and police. Interventions aimed at these groups were implemented by NGOs, with funding coming from bilateral partners and from the National AIDS Council, whose Common Fund was financed by the Bank until 2009.
  • The following interventions took place among commercial sex workers in provinces prioritized because they had high HIV prevalence rates and/or were major transport corridors or areas where major investment projects existed (coal extraction, major road/bridge construction, natural gas).
    YearNumber of contacts between peer educators and CSWNumber of information, education, and communication materials distributedNumber of male condoms distributedNumber of female condoms distributed
    200934,67835,548278,28520,074
    201052,98888,251887,65883,755
    201140,94013,331352,25542,007
  • In 2011, nearly 5,000 men who gave sex with men were reached by prevention interventions including condom distribution, peer education, and provision of water-based gel. Clinical referral services were provided to nearly 500 individuals.
  • In 2011, 7,200 long-distance truckers on main transport corridors were reached with condom distribution, counseling, promotion of HIV testing, and treatment of sexually transmitted infections. In 2008, 33,446 truckers received information on HIV prevention.
  • In 2011, 23,364 prisoners were reached with face-to-face counseling, distribution of information, education, and communication materials, peer education, and/or referral to health services.

Outcomes:

The ICR provides information on the following indicators:

  • The percentage of 15-24-year olds who had sex before age 15 was 27.7% in 2006, and 25% for females and 24.8% for males in 2011, against a target of less than 20%.
  • The percentage of 15-24-year olds who had sex with a non-marital, non-cohabiting partner in the last 12 months declined from 60% in 2002 to 4.2% for females and 16.4% for males in 2011, against a target of 50%.
  • The percentage of 15-24-year olds who used a condom the last time they had sex with a non-regular partner was 31% in 2002, and 33.1% for females and 37.2% for males in 2011, against a target of 37.2%.

In reporting on these indicators, the ICR relies on source material that should have been scrutinized more carefully, and it does not seek verification of baseline and endline data points that do not seem reasonable or that seem contradictory with one another. Also, more reliable and comparable data are readily available. Mozambique Demographic and Health Surveys (DHS) and AIDS Surveys from 2003 and 2009 show the following:
  • The percentage of females who know that using a condom prevents HIV transmission increased from 53% to 71%, and men from 71% to 74%. The percentage of females who know that having one exclusive sexual partner prevents transmission of HIV increased from 53% to 73%, and males from 73% to 76%. This is very limited improvement among men, and indicates that 25-30% of adults remain unaware of basic facts about HIV transmission.
  • The percentage of 15-19-year-olds who had sex before age 15 has declined slightly (from 28 to 23% for girls, and 31 to 27% for boys). The percent of never-married 15-19-year-olds who had sex in the past 12 months was nearly unchanged for women (from 48 to 47%) and declined slightly for men (from 61 to 55%). The percent of this group who used condoms in the last sexual encounter increased but was still less than half, from 34 to 45% for women, and 31 to 37% for men.
  • The percent of men who paid for sex in the past 12 months declined only slightly (from 13 to 9%) and condom use in last paid sex was still low at 28%, compared with 21% in 2003.
  • Among adults with 2 or more partners, condom use improved but was still was still low: 24% for women (up from 14% in 2003) and 22% for men (up from 19% in 2003).

These results show modest change in some areas, but levels of knowledge and behavior that indicate still-high risk of spread of infection. The ICR also reports no data on many high-risk groups most likely to spread HIV infection.

The project team later added that the percentage of HIV-positive pregnant women receiving PPTCT services increased from 14.2% in 2006 to 66.4% in 2011.

Mitigate the effects of the epidemic (Original and Revised Objective) is rated Substantial.

Outputs:

The Bank’s Treatment Acceleration Program (US$ 59.8 million, 2004-2008) had a Mozambique component, and therefore the project under review here was designed only to fund procurement of drugs for OIs, PPTCT, and post-exposure prophylaxis. However, the dramatic price drop for ARV allowed procurement of ARV for general purposes under the HARP project, albeit on a small scale. The ICR (p. 30) states that ART is now available in all districts, but it does not provide information on what treatment was specifically provided through the HARP project.

Of the 3,771 funded community-based subprojects, according to the ICR (p. 24), “many focused on mitigation of the impact of HIV infection and thus turned into income-generating projects.” Over 20% of the community-based subprojects “dealt with orphans” (ICR, p. 14). However, no further information is provided in the ICR. The project team later added that the specific activities implemented in all provinces of the country through the community-based subprojects included home-based care, provision of nutrition support and school materials for orphans and vulnerable children (OVCs), and provision of seed money to initiate income-generating activities to families (hammer mills, chicken rearing, small-scale farming, tailoring). According to this additional information, in 2007 455 subprojects supported 428,299 orphans and vulnerable children (an estimated one-third of OVCs in the country), 28,845 people living with HIV/AIDS, and 19,191 households (foster families, and households headed by the elderly).

The ICR (p. 28) states that, through the public sector component, the Ministry of Women and Coordination of Social Action supported orphans and vulnerable children and income-generating projects for women affected by HIV, but it does not provide details.

Outcomes:

In 2011, 96.2% of HIV-infected tuberculosis patients received cotrimoxazole, against a target of 100%. In 2011, 10% of HIV-infected persons received isoniazide preventive therapy, against a target of 40%.

The project team provided additional information that the coverage of those eligible for treatment with ARV increased from less than 5% n 2004-2005 to 40% (nearly 290,000 individuals) in 2012. According to the project team, the main contribution of the HARP project was not directly in the procurement of ARV, but in the assistance provided to strengthen laboratory services through procurement of HIV tests, syphilis tests, and general equipment fo improve biosafety, and through the training of staff that provide clinical and preventive services throughout the country.


5. Efficiency:

Efficiency is rated Substantial.

The PAD (pp. 26-28, 80-81) cites an economic analysis specific to Mozambique performed in mid-2001, showing an economic loss in the country due to HIV/AIDS of between US$11 million and US$ 37 million in 2002, rising gradually in subsequent years as the population and GDP increase. These losses would stem from reduced productivity growth, reduced population growth and human capital accumulation, and reduced physical capital accumulation. This analysis also indicated that successful HIV prevention policies could significantly impact economic growth, justifying large expenditures on prevention. However, there is little evidence on whether the project was as effective as the assumptions made in the PAD; a fresh economic analysis was not carried out for the ICR.

According to the project team, there were no overruns in any category of expenditure of the project. In addition, despite the project having been extended nearly three years beyond its original closing date, administrative costs were not substantially increased, in part because of the merging of project funds with the National AIDS Council Common Fund.

The ICR (p. 12) states that the community-based subprojects focused on alleviating the difficult conditions of people living with HIV and AIDS “showed how a small amount of money can go a long way” to provide relief to needy individuals and communities, but it provides no further information about the content of these subprojects or their cost-effectiveness. The project later provided information on the activities funded by the subprojects, but still no analysis of their cost-effectiveness.

The financial management procedures put in place proved to be overly complicated, as two special accounts (one each for the Ministry of Health and the National AIDS Council) resulted in 33 project accounts at the provincial level. Payment procedures were too complex for the provinces and the small community organizations that were sometimes unable to open a bank account. This complexity resulted in delays in implementation of community-based subprojects.

The ICR acknowledges (p. 12) that efficiency would have been enhanced through a greater focus on geographic areas and population groups with the highest risk of transmitting HIV infection. Efficiency of the public sector component was diluted by the large number of ministries involved (ICR, p. 13). After the mid-2009 restructuring, efficient targeting of resources improved through the selection of larger organizations to implement community-based subprojects and by the targeting of some of those subprojects in areas most likely to prevent the spread of infection (multiple concurrent partnerships, male circumcision).

Integration of HIV and TB services during the latter half of the project led to efficiency gains, though the ICR does not provide evidence to demonstrate that this was the case. The project team stated that these efficiency claims are based on the cross-screening of patients. Screening is no longer done along vertical lines, so that a TB worker now knows that HIV status should be checked, and vice versa. The project team later provided additional information from the 2009 UNGASS report on the integration of HIV and TB services, indicating an increase over time in the percentage of TB patients screened for HIV and vice versa. This increase, according to the project team, stemmed largely from a 2008 Government decision to scale up screening of TB among HIV positive patients receiving care in health facilities.


Year% of TB patients HIV tested% of TB patients HIV+% HIV + TB patients receiving cotrimoxazole% TB patients on ARV therapy
20076847.39333
200879609230
200984668922

YearNumber of HIV+ individuals screened for TBNumber of HIV+ individuals diagnosed with TBNumber of HIV+ individuals provided with Isoniazid prophylaxis
20073,0002,265676
20084,880439724
200924,3302,1182,429


a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Under both the original and revised objectives, the project’s objectives and design were substantially relevant, and achievement of one development objective and efficiency were substantial, but achievement of the other development objective was modest. There is no evidence that the project's efforts to reach the high-risk groups most likely to spread HIV infection resulted in increased knowledge or behavior change among those groups, and available evidence on slowing the spread of the epidemic indicates modest progress. The outcome rating under both the original and revised objectives is therefore Moderately Satisfactory.*

*Under the IEG/OPCS Harmonized Evaluation Criteria, when the operation’s objectives have been formally changed through restructuring, the outcome rating is obtained by assessing the project based on the original objectives over the entire life of the project, assessing the project based on the revised objectives over the entire life of the project, and calculating the weighted average of the two outcome ratings according to the share of total Bank disbursements at the time of the restructuring. In this case the outcome rating for both the original and the revised objectives is the same, so weighting will not change the result.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

Because of the large number of other donors contributing to HIV/AIDS activities, the Bank decided against a follow-on AIDS operation, instead focusing post-HARP project support on health sector reform. The Bank has continued to use its influence with the Ministry of Finance to keep HIV as a priority at the macroeconomic level. Nonetheless, due to the ongoing international financial crisis, development assistance for AIDS from bilateral and multilateral sources may decline at the very moment that resource needs (especially for ART) are increasing. There is also a concern that, without the Bank as a part of the pooling arrangement, fiduciary oversight may suffer.


Institutional capacity development at CNCS appears likely to be sustained, as does similar capacity development achieved by the project for the Provincial Nucleus for AIDS Control (NPCS), civil society organizations, NGOs, and public sector agencies.

The private sector activities in Component 1 have evolved into a self-standing initiative led by the private sector itself, ECoSida, with about 50 active members. CNCS has continued to meet with ECoSida, mobilizing managers and facilitating workplace interventions.

Without behavior change interventions focusing on the high-risk and vulnerable groups most likely to transmit infection, new infections will not be prevented effectively and efficiently.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:
The operation was prepared on a fast-track basis, and as a result some unresolved actions were pushed forward to implementation (in particular, numerous conditions for effectiveness and resulting delay in effectiveness) (ICR, p. 16). Lessons from earlier MAP projects were incorporated into project design, particularly the need for appropriate procedures to engage civil society organizations (PAD, pp. 14-15), and the role of a national AIDS council as coordinator rather than controller or implementer. However, some important lessons from earlier experience were not sufficiently considered, including the need for robust monitoring and evaluation; the ICR (p. 5) states that insufficient attention to M&E during preparation led to weaknesses in the indicators in the results framework (see Section 10). While capacity-building for civil society organizations was recognized as a priority, insufficient preparatory work was done in this area, and fiduciary, cost-sharing, and reporting requirements were inappropriately burdensome for rural communities and civil society organizations. The preparation team may also have been too optimistic in its institutional assessment of the CNCS, in terms of its ability both to monitor subprojects and to coordinate the overall national response (PAD, pp. 41-43; ICR, p. 16). Also according to the Bank's Operations Portal, the project had a US$ 880,000 preparation facility that was never disbursed; the ICR does not mention this facility or explain why it was not used. The project team clarified that the preparation facility was fully disbursed.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
The PAD (pp. 110-111) contained a Supervision Plan outlining the strategic areas to receive focus during the project’s first year: the comprehensiveness of the approach, engagement and ownership-building among stakeholders, capacity-building in the CNCS, and strengthening of the fiduciary architecture. The task team leaders and experienced fiduciary management staff were field based, and intense and frequent supervision made possible the various restructurings and amendments in response to implementation issues. However, it took a long time to finalize some implementation arrangements (16 months for internal processing of the Common Fund scheme, for example). The project team and the Bank’s Global AIDS Monitoring and Evaluation Team (GAMET) provided “considerable assistance” to the program (ICR, p. 16), but reporting was still not well done, and only the final supervision report contained a table with the relevant indicators. Development partners relied on the Bank team for fiduciary oversight after the Bank joined the pooling arrangement. Although a comprehensive Environmental Plan was developed to deal with medical waste in public sector clinics, private sector clinics were not addressed, and “the ambitious plan was not systematically implemented” (ICR, p. 17). “During implementation, monitoring of environmental arrangements and compliance was virtually nonexistent, as not a single mission was joined by an environmental specialist, and none of the supervision reports paid any significant attention to these aspects” (ICR, p. 17). However, compliance with environmental requirements was consistently rated as “satisfactory.” The Borrower commented (Summary of the Borrower’s ICR, p. 41) that there was “deficient and unclear” communication from the Bank about criteria for rating various aspects of project and subproject performance, making it difficult to take adequate corrective measures.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
Political commitment to the fight against HIV/AIDS in Mozambique was high from the start of the project, with consistent public engagement at the level of the President and Prime Minister. According to the ICR (p. 18), the Ministry of Health performed well, with achievement of the health sector component “well beyond its targets” and achieving targets for VCT and ART; however, the ICR does not provide specific information about provision of ART. The performance of non-health ministries varied, with true mainstreaming absent in most ministries. Focal points, at the central and provincial levels, were part-timers and did not receive additional payments to support mainstreaming. However, with the creation in 2007 of a new Civil Service Ministry, an HIV/AIDS strategy for civil servants was developed, and funds were received to target prevention activities in 12 priority ministries (the ICR does not specify which 12).

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
The main implementing agencies were the Ministry of Health and National AIDS Council, including a project unit (Office of Coordination and Investment Projects) in the MOH. Civil society organizations and NGOs as subproject implementers, as a whole, despite their high level of motivation and eagerness to comply with the project’s requirements, lacked the necessary implementation capacity, and CNCS was not capable of managing thousands of sub-projects, causing delays in approvals and replenishments. Fiduciary requirements were heavy for rural communities where banking was virtually non-existent and multiple bids hard to obtain. Reporting and cost-sharing requirements were difficult or prohibitive for many organizations. Some of these issues were rectified during the life of the project; for example, UNAIDS, PEPFAR, and others provided capacity-building to civil society organizations and technical assistance to CNCS. Following the 2007 restructuring, a Grant Management Agency (GMA) was contracted to handle the selection of subprojects; this was problematic in implementation, as CNCS management did not agree to relinquish those responsibilities, and as a result the GMA resigned after one year. The CNCS continued to handle subproject selection functions until the end of 2009, diverting its attention from its main responsibility of coordinating the national response. However, the ICR (p. 7) does note that CNCS provided “considerable” training to implementing agencies, journalists, youth groups, and CNCS sub-structures at the provincial level. From the end of 2009 until project closure, UNICEF effectively managed the contracting of community-based subprojects.

The agency hired to implement the private sector component performed well, exceeding subproject targets through a focus on workers in general, workers living with HIV and their families, the neighborhoods in which they are located, and orphans of deceased workers.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
According to the ICR (p. 7), the project’s M&E architecture and staffing were weakly developed during preparation, and the PAD’s M&E discussion (p. 19) acknowledged that some components of the M&E framework were significantly more robust than others. Appropriate attention was paid to the establishment of databases at the national and provincial levels for fiduciary matters, but this prioritization may have taken attention away from collection of core epidemiological and behavioral data at baseline.

The project’s original key outcome/impact indicators focused on HIV prevalence among 15-24-year-old pregnant women, the age of sexual debut, the number of sexual partners outside of primary union in the last 12 months among persons ages 15-49, and the rate of condom use in sexual encounters outside of the primary relationship in the last 12 months among persons ages 15-19. None of these indicators captured the objective of mitigating the effects of the epidemic through care activities. When the project was restructured in 2007, the indicator related to HIV prevalence was appropriately dropped, but no indicators were added to capture the achievement of the mitigation objective.

b. M&E Implementation:
Three M&E working groups were established: one led by CNCS to facilitate dialogue around strategic planning, M&E policy, and technical issues; one led by MOH to focus on epidemiology and health service statistics; and a third broadly-based group to focus on the multi-sectoral response. Early planned technical assistance for M&E was not hired; “this hampered especially CNCS’s role to collect and analyze periodic reports of implementers, and prepare annual implementation reports on the national response” (ICR, p. 7). It is not clear why this planned technical assistance did not materialize. With external support from the UN, a National M&E Framework was developed and published in 2006 in the context of the preparation of the second phase of the National HIV/AIDS Strategic Plan. CNCS also received support from the Bank’s GAMET in 2005-2006. However, in 2007 CNCS still lacked dedicated information technology specialists for programming, database design and maintenance, and computer-based communications. Eventually the United States Agency for International Development (USAID) funded technical assistance in this area. After 2007, CNCS gained increasing capacity to report reliably on about two-thirds of the M&E indicators in the National Framework, and to report on grants made and amounts disbursed for subprojects. CNCS officers made regular visits to the provinces, and M&E officers were eventually hired at the provincial level. By project closure, however, CNCS’s data sets were still incomplete, with organizations executing subprojects the only entities that reported regularly; many donors and other implementing agencies did not report systematically. During the project’s lifetime, the Bank contributed to the implementation of some surveys that provided information on knowledge, attitudes and practices, as well as behavioral and epidemiological trends (2004 Demographic and Health Survey; 2009 National Survey on Prevalence, Risks, Behavior and Information).

a. M&E Utilization:
CNCS has published annual reports since 2001, and these reports have become more comprehensive over time. A technical working group on M&E, jointly chaired by the MOH and CNCS and widely attended by many stakeholders, analyzes and draws conclusions from the data. These discussions, according to the ICR (p. 8), have helped to focus attention toward prevention, including the need to reduce concurrent sexual partnerships. Data sharing among development partners, however, remains problematic, and CNCS is reviewing options to encourage data sharing.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
This Category B project triggered the Environmental Assessment (OP/BP/GP 4.01) safeguard policy. The most important environmental issue was management of medical waste. Although a comprehensive Environmental Plan was developed to deal with medical waste in public sector clinics (with components to strengthen the regulatory framework, increase capacity among health care and waste management workers, increase public awareness, provide improved equipment for collecting and sorting waste in public health care facilities, and install modern incinerators where appropriate), private sector clinics were not addressed, and “the ambitious plan was not systematically implemented” (ICR, p. 17). “During implementation, monitoring of environmental arrangements and compliance was virtually nonexistent, as not a single mission was joined by an environmental specialist, and none of the supervision reports paid any significant attention to these aspects” (ICR, p. 17). However, compliance with environmental requirements was consistently rated as “satisfactory.” The project team explained that there was a health care waste management plan that was fully implemented by MOH. According to the project team, when health facilities were visited during missions, it was clear that the plan was being implemented: there were sharps containers and protections for health workers. The project team states that it was therefore unnecessary to have an environmental specialist on missions to verify compliance.

b. Fiduciary Compliance:
Primarily with USAID support, an accounting system was put in place to respond to the needs of Government and donors, tracking donations and use of funds at the central and provincial levels. The financial management procedures put in place proved to be overly complicated, however, as two special accounts (one each for MOH and CNCS) resulted in 33 project accounts at the provincial level. Payment procedures were too complex for the provinces and the small community organizations that were sometimes unable to open a bank account. This complexity resulted in delays in implementation of community-based subprojects. In some cases, the provincial nuclei of CNCS managed finance for community-based organizations on their behalf, creating a potential conflict of interest. The 20% contribution required from Government hampered implementation and added to the complexity of the procedures, as it required special bank accounts; the 20% contribution was abandoned in 2007. According to the ICR, financial management arrangements and compliance were well supervised, and issues raised during missions and audits were systematically and rigorously followed up. The ICR does not state whether audits were on time and unqualified. The project team stated that all audits were on time and unqualified.


In 2007, the Bank joined a “Common Fund” arrangement that pooled contributions with other multilateral and bilateral partners. The RRF was also financed together with other donors. After the Bank joined these pooling arrangements, development partners began to rely on the Bank for fiduciary oversight.

The CNCS Secretariat was charged with procurement oversight for the entire Program, and it hired a procurement officer for this task. MOH handled large procurements and all packages specific to Component 4. Procurement plans of good quality were regularly sent to the Bank, but record keeping was not always in good order (ICR, p. 9). Procurement for the community-based subprojects was done by CNCS, but this task was beyond its capacity, and it pulled the main area of CNCS focus away from its primary function to coordinate the national response. The ICR reports no cases of misprocurement.

c. Unintended Impacts (positive or negative):
None.

d. Other:
None.



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Significant
Significant
 
Bank Performance:
Moderately Satisfactory
Moderately Unsatisfactory
Insufficient attention to M&E during preparation led to weaknesses in the indicators in the results framework. While capacity-building for civil society organizations was recognized as a priority, insufficient preparatory work was done in this area, and fiduciary, cost-sharing, and reporting requirements were inappropriately burdensome for rural communities and civil society organizations. The preparation team was too optimistic in its institutional assessment of the CNCS, in terms of its ability both to monitor subprojects and to coordinate the overall national response. It took a long time to finalize some implementation arrangements, and reporting was not well done. During implementation, monitoring of environmental arrangements and compliance was inadequate. 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Unsatisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The following lessons are drawn largely from the ICR (pp. 19-20), with adaptation:

Project development objectives must state clearly what an individual project is doing and for what it is to be held accountable, with a concise set of performance indicators to match. In this case, the objectives stated in the PAD and the DGA were not consonant, and key performance indicators – even after restructuring – did not capture progress on one of the two main development objectives.

Projects should be tailored to institutional and community capacity, and careful capacity assessments should point the way toward capacity development where skills and resources are insufficient. In this case, lack of readiness, difficulties in mastering Bank procedures, and insufficient recognition of the need for intensive capacity building up front and during early implementation led to delays and inefficiencies. The need to estimate accurately the capacity development needs of civil society organizations in particular cannot be overstated. In addition, careful community infrastructure assessments should reveal and correct mismatches between Bank requirements and realities on the ground, as in this project where banking transactions were required in environments where banks did not exist.

Targeted approaches, tailored to each individual country’s HIV epidemic, are essential for success in preventing the spread of infection. Prioritization of prevention efforts targeted at high-risk or vulnerable groups most likely to spread infection is essential for both efficacy and efficiency – either as a goal in itself, or, as the ICR (p. 20) suggests, as part of a broader strategy of capturing the energy of newly-empowered community organizations.

IEG offers the following additional lessons:

HIV/AIDS projects must prioritize not only activities that are most effective at slowing the spread of the epidemic, but also the implementers who are most effective in delivering these activities. In this case, the Ministry of Health did have the capacity and expertise to implement what was expected, while the performance of many other agencies or ministries was more challenging, and the types of activities sponsored were not necessarily the highest priority for slowing the spread of HIV.

Where appropriate, ICRs should take into account data and evidence from sources beyond a project's M&E system. In order to demonstrate the achievement of objectives, it may be necessary to tap all available information -- not limited to the indicators in a project's results framework -- including data sources from outside the project and to triangulate the results. This is all the more important when there are shortcomings in the design of key indicators and other donors are active in the same sector.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR presents a clear account of the course of project implementation. However, it provides much more information on process than on outputs and outcomes, making it difficult to assess achievement of objectives. It does not critically assess key data points, and it does not take into account readily available additional data to demonstrate progress in achievement of the project's objectives. Important information on the project's costs, financing, and Borrower contribution are contradictory or missing.

a. Quality of ICR Rating: Unsatisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions