Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Multi-sectoral HIV/AIDS Program


  
1. Project Data:   
ICR Review Date Posted:
01/09/2013   
Country:
Ghana
PROJ ID:
P088797
Appraisal
Actual
Project Name:
Multi-sectoral HIV/AIDS Program
Project Costs(US $M)
 35.3  26.13
L/C Number:
C4125
Loan/Credit (US $M)
 20.0  14.53
Sector Board:
Health, Nutrition and Population
Cofinancing (US $M)
 8.3  8.3
Cofinanciers:
DfID
Board Approval Date
  11/15/2005
 
 
Closing Date
06/30/2011 06/30/2011
Sector(s):
Central government administration (47%), Health (31%), Other social services (12%), Sub-national government administration (10%)
Theme(s):
HIV/AIDS (33% - P) Population and reproductive health (17% - S) Health system performance (17% - S) Participation and civic engagement (17% - S) Gender (16% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Judyth L. Twigg
Robert Mark Lacey Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:


    According to the Development Credit Agreement (DCA, p. 15) and the Project Appraisal Document (PAD, p. 7), the project’s original objective were “to support the Borrower’s Program to: (i) reduce the new [HIV] infection rate among vulnerable groups and the general population; (ii) mitigate the impact of the epidemic on the health and socio-economic systems as well as infected and affected persons; and (iii) promote healthy lifestyles, especially in the area of sexual and reproductive health.”

    The revised objectives of July 3, 2007, according to the Proposed Umbrella Restructuring and Amendment of the Financing Agreements for the Projects under the Multi-Country HIV/AIDS Program for Africa (MAP) (p. 11), were: “to increase access to HIV care, prevention, and impact mitigation services for high-risk groups, vulnerable populations, and the general population.”

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 07/03/2007

c. Components:

The project’s seven components mirrored the seven goals of Ghana’s National HIV/AIDS Strategic Framework 2006-2010 (NSF II):

1. Policy, Advocacy, and Enabling Environment (appraisal, US$ 4.0 million; actual, US$ 0.58 million). This component was to support reviewing, formulating, and enforcing policies to protect the rights of persons living with HIV/AIDS (PLWHA) and/or people affected by HIV/AIDS, including creating an enabling environment and carrying out activities to empower and assist PLWHA and orphans and vulnerable children (OVC) and decreasing vulnerability of target groups to infection through support and advocacy. This component was also to support coordination and implementation of the national response to HIV/AIDS.

2. Coordination and Management of a Decentralized Multi-Sectoral Response (appraisal, US$ 6.8 million; actual, US$ 4.15 million). This component was to support coordination and management of the ministries, departments, and agencies, Regional AIDS Committees, Metropolitan Municipal Assemblies, and District AIDS Committees in implementing their programs. Activities included defining roles and responsibilities at the central, regional, district, and community levels; supporting key agencies to develop, implement, and monitor activities; and helping to develop HIV/AIDS sectoral plans, workplace interventions, and staff training. This component was to include advocacy to increase political commitment through sensitization programs, development of annual work plans, and improved information flows at all levels.

3. Mitigating Economic, Socio-Cultural, and Legal Impacts (appraisal, US$ 2.6 million; actual, US$ 2.33 million). This component was to support activities to mitigate the impact of the HIV/AIDS epidemic on vulnerable social groups, including education support and training for OVCs, development of income-generating activities for vulnerable households, provision of community-based and institutional care for orphans, and provision of psycho-social and legal support for affected families.

4. Prevention and Behavior Change Communication (appraisal, US$ 3.2 million; actual, US$ 6.0 million). This component was to develop and implement activities aimed at bringing about behavioral change among at-risk groups, including in-school youth and mobile populations. Programs were to include promoting the use of quality services for sexually transmitted infections (STIs) and voluntary counseling and testing (VCT), preventing mother-to-child transmission (PMTCT), minimizing the risk of HIV transmission through blood and blood products, promoting condom use during higher-risk sexual encounters, and reducing occupational exposure to HIV and other infections. Partnerships with private enterprises were to be encouraged in implementing these programs.

5. Treatment, Care, and Support (appraisal, US$ 2.2 million; actual, US$ 0.06 million). This component was to scale up antiretroviral therapy (ART), VCT, management of STIs and opportunistic infections, home-based care, and community care and support for PLWHA, affected families, and OVCs.

6. Research, Surveillance, and Monitoring and Evaluation (M&E) (appraisal, US$ 1.2 million; actual US$ 1.06 million). This component was to carry out activities to enable effective assessment of the HIV/AIDS epidemic and national response, including biological and behavioral surveys, poverty analyses, priority specialized studies, and program activity and financial monitoring.

7. Resource Mobilization and Funding Arrangements (appraisal, US$ 0.0 million; actual, US$ 0.353 million). This component was to establish a flexible fund arrangement for mobilizing and channeling HIV/AIDS resources for the national response; streamlining the budget process for HIV/AIDS resources; establishing an effective targeting and resource allocation process; strengthening financial management and monitoring; and carrying out procurement and financial audits.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Cost: At the time of restructuring, US$ 3.69 million, or 18.45% of the loan, had been disbursed.

Estimated and actual costs by component were available only for the IDA credit, not for the entire project. The costs cited in Section 2c and discussed below are therefore for the IDA credit only.

The ICR does not explain why significantly less was spent than planned on Components 1, 2, and, 5, and more than planned spent on Component 4. It also does not provide final total project costs. The project team explained that spending was less than planned for Component 1 because policy development capacity was weak, and because there was other donor funding in this area. Spending was less than planned for Component 2 and more than planned for Component 4 because of changes after the mid-term review (MTR) that shifted focus away from treatment and toward prevention, and because of funding from other donors for treatment. The project team stated that final total project costs were the sum of the actual Bank contribution, the DFiD contribution, and the Borrower's contribution.

Financing: The estimated project cost of US$ 35.3 million was expected, over the course of its lifetime, to finance 6.1% of Ghana’s National HIV/AIDS Strategic Framework 2006-2010.

This was a repeater operation of the Bank’s Ghana AIDS Response Project (GARFUND, US$ 27.8 million, 2001-2005).

US$ 4.6 million of the Bank loan was cancelled in June 2010, based on the recommendations made by a joint Bank-UNAIDS mission that took place in June 2009. Following that initial cancellation, the project team assessed the Borrower’s proposals for activities to be funded with the remainder of the loan, and found that some of these activities did not meet agreed disbursement conditions. As a result, another US$ 0.6 million of the loan was cancelled.

The project included common implementation arrangements with the Department for International Development (UK, DfID) by pooling funds in an account managed by the Ghana AIDS Commission (GAC).

Borrower Contribution: The ICR does not state how much of a planned US$ 7.0 million contribution was actually made by the Borrower. The project team confirmed that US$ 3.3 million was contributed.

Dates: The project was restructured on July 3, 2007, as part of an umbrella restructuring of the Multi-Country AIDS Program for Africa (MAP) projects. It closed on schedule.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Original objectives: Substantial. HIV prevalence rates in Ghana had increased in the years before appraisal, from 2.6% in 2000 to 3.1% in 2004. Considerable variations existed by geographic region, gender, age, occupation, and urban-rural residence. Behaviors associated with the spread of HIV infection, such as higher-risk sex (defined as more than one partner in the last 12 months), were common, especially among young people. The project’s objectives were substantially relevant to Ghana’s National Strategic Framework II (2006-2010), from which the project directly took its development objectives and components. The objectives are also substantially relevant to the Bank’s Country Assistance Strategy (CAS, 2008-2011), current at closure, which contains a focus on strengthening reproductive health care and linking that care to HIV/AIDS control, as well as a focus on HIV prevention as a contributor to human resource development.

Revised objectives: Substantial. The objectives remained substantially relevant to country conditions, the national HIV/AIDS strategy, and Bank strategy. The restructuring rephrased two of the three original objectives into a single objective, still maintaining the broad goals of both HIV prevention and impact mitigation among both vulnerable groups and the general population. The removal of the third original objective related to healthy lifestyles produced somewhat greater coherence to the objectives.

b. Relevance of Design:

Relevance of design to the original objectives: Modest. According to the ICR (p. 13), despite being designed as a Specific Investment Loan, the project adopted the country National Strategic Framework as its own framework and therefore took on an implicit Adaptable Program Loan Approach. Tranches of funds were released on a regular basis, without agreed milestones, and no specific plans were made for procurement of goods or services. The ICR (p. 13) states that “this design ambiguity contributed to difficulty in implementation.” A related shortcoming is the broadness of the statement of objectives, which, according to the ICR (p. 19), “may have discouraged the development of a coherent results chain, in which the project’s inputs and activities would be more clearly linked with the stated objectives.” Together with the large number of components, the broad objectives failed to specify priority areas for intervention. There was no clear linkage between the objectives, the project’s expected outcomes as measured by the performance indicators, and the activities to be financed. Subproject grants were too small, geographically spread out, fragmented, and intermittent to be reasonably expected to produce the anticipated outcomes.

Relevance of design to the revised objectives: Modest. Project design did not change significantly as a result of the restructuring. One improvement reduced the number of subproject grants to be financed (from almost 2,000 in 2006 to 30 in 2009), through umbrella organizations that were more likely to deliver implementable and monitorable projects that could reasonably be expected to produce desired project outcomes.


4. Achievement of Objectives (Efficacy) :

This project occurred simultaneously with the Bank’s Abidjan-Lagos Transport Corridor Project (an HIV/AIDS prevention/treatment/mitigation project, US$ 12.9 million, 2003-2007) and Treatment Acceleration Project (US$ 14.9 million, 2004-2007), as well as six different Global Fund HIV/AIDS grants (US$ 14.2 million, Round 1, 2003-2007; US$ 114.0 million, Round 5, 2006-2013; US$ 4.9 million, Round 8, 2010-2012; US$ 2.9 million, Round 8, 2010-2012; US$ 35.9 million, Round 8, 2010-2012; and US$ 13.8 million, Round 8, 2010-2011). The ICR notes that the "tenuous links between project inputs and measurable social development outcomes" made attribution "extremely difficult" (p. 24). In particular, the ICR (p. 18) states that provision of antiretroviral therapy to adults and youth and provision of services to prevent HIV transmission from mothers to children were funded by the MOH and other donors, not by the project, and so reported outcomes in these areas are not relevant to an assessment of the project. The ICR provides very little information on the project's activities (outputs), making it impossible to construct a complete results chain. Attribution is further complicated by the lack of data for the years specific to the project; baseline data for most indicators are drawn from the 2003 Demographic and Health Survey, conducted two years before the project became effective.

Original objectives

Reduce the new HIV infection rate among vulnerable groups is rated Negligible. The ICR provides very little information on the project's activities (outputs) relevant to this objective, making it impossible to attribute observed outcomes to project-financed interventions and therefore construct a complete results chain. No data on changes in risky behavior are provided for key vulnerable groups.

Outputs:

Between 16 and 22 ministries, departments, and/or agencies annually implemented sector strategic plans related to HIV/AIDS. The percentage of sector ministries with HIV/AIDS work plans and budgets approved and funded by the Ghana AIDS Commission increased from 17% in 2006 to 34.6% in 2009. Several hundred staff received training on procurement, financial management, project management, and data management. The execution rate of total HIV/AIDS allocations by key ministries, departments, and agencies increased from 70% in 2005 to 99.5% in 2008/2009, exceeding the target of 90%.

1,975 subprojects were funded. The ICR reports that these subprojects reached an annual number of beneficiaries peaking at 30.4 million in 2008, but these data are of questionable meaning, since the population of Ghana was 24.4 million in 2010. The ICR (p. 11) speculates that these data, provided by the Ghana AIDS Commission, repeat-count estimates of the general population reached by mass media messages. The ICR does not provide information on the content of these subprojects, beyond general conclusions from an end-of-project beneficiary survey that the project contributed to empowerment of women (including empowerment to insist on condom use), increased HIV knowledge and awareness among women and youth, and provided unspecified counseling and testing services. The provision of condoms by the project is also implied in the beneficiary survey analysis, but no specific information is provided.

The percentage of total national HIV/AIDS funds spent on prevention activities for selected vulnerable groups (commercial sex workers, men having sex with men, prisoners, member of the military, and OVCs) increased from 0.4% in 2003 to 3.8% in 2011, not meeting the target of 10%. No information is provided on the content, coverage, or effectiveness of prevention interventions targeted at these populations.

Outcomes:

The percentage of female youth reporting the use of a condom during higher-risk sex increased from 32% in 2003 to 45% in 2011, exceeding the target of 40%. The percentage of male youth reporting the use of a condom during higher-risk sex decreased from 52% in 2003 to 46% in 2011, not meeting the target of 46%. Baseline data are from a 2003 DHS Survey, and endline data are from a 2001 end-of-project survey. According to the ICR, data are not available for the years specific to the project.

No information is provided on trends in risk behavior among high-risk groups such as commercial sex workers, men having sex with men, prisoners, or members of the military.

Reduce the new HIV infection rate among the general population is rated Negligible. The ICR provides very little information on the project's activities (outputs) relevant to this objective, making it impossible to attribute observed outcomes to project-financed interventions and therefore construct a complete results chain.

Outputs:

In addition to the output data above, 92.1% of all teachers were trained in life skills-based HIV education and taught those skills during the 2009/2010 academic year. 79.1% of all schools (84.0% of public and 59.9% of private) have an HIV Alert Program in 2009. The ICR does not provide data on prior years, so no trends can be observed.

Outcomes:

Median age at first sex increased from 18.3 years for females in 2003 to 19 years in 2011, but decreased from 20.2 years for males in 2003 to 19 years in 2011. The target of 19 years for females was met, but the target of 20.5 years for males was not met. According to the ICR, data are not available for the years specific to the project.

The percentage of women aged 15-49 who had higher-risk sex in the past 12 months decreased from 21% in 2003 for females to 15.8% in 2011 (DHS data), meeting the target. The percentage of men aged 15-49 who had higher-risk sex in the past 12 months decreased from 38% in 2003 to 26% in 2011, meeting the target. According to the ICR, data are not available for the years specific to the project.

The percentage of adult females reporting the use of a condom during higher-risk sex increased from 28% in 2003 to 39% in 2011, not meeting the target of 42%. The percentage of adult men reporting the use of a condom during higher-risk sex increased from 44.8% in 2003 to 53% in 2011, not meeting the target of 65%. According to the ICR, data are not available for the years specific to the project.

Mitigate the impact of the epidemic on health and socio-economic systems is rated Negligible.

The ICR does not provide data on activities or outcomes related to this objective.

Mitigate the impact of the epidemic on infected and affected persons is rated Negligible. The ICR does not provide output data that would make possible attribution of observed outcomes to the project.

The ratio of current school attendance among orphans to that among non-orphans (ages 10-14) remained unchanged at 0.8 between 2003 and 2008, not meeting the target of 0.9. The ICR does not provide more recent data for this indicator.

Promote healthy lifestyles, especially in the area of sexual and reproductive health is rated Negligible.

The ICR does not provide data on activities or outcomes related to this objective.

Revised objectives

Increase access to HIV care services for high-risk groups, vulnerable populations, and the general population is rated Negligible, based on the data on presented above.

Increase access to HIV prevention services for high-risk groups and vulnerable populations is rated Negligible, based on the data presented above.

Increase access to HIV prevention services for the general population is rated Negligible, based on the data presented above.

Increase access to impact mitigation services for high-risk groups, vulnerable populations, and the general population is rated Negligible, based on the data presented above.


5. Efficiency:


Efficiency is rated Negligible.

No formal economic or financial analysis was conducted at appraisal.

The ICR provides very little analysis of or information on efficiency. It observes (p. 23) that the project did not carry out geographic targeting of the regions of Ghana with highest HIV prevalence, where spending could have had the greatest impact. The project did not effectively prioritize the MOH, with funds covering a large number of ministries and agencies, and according to the ICR (p. 20), the MOH was excluded from many interventions where its participation was essential. Also, the project did not effectively prioritize prevention interventions among groups at highest risk of transmitting infection. The awarding of many small subproject grants in the early years of implementation raised transaction costs and reduced efficiency; this was corrected at mid-term, when the annual number of grants was dramatically reduced and responsibility for engaging small civil society organizations was given to larger umbrella groups.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Original objectives: Based on Substantial relevance of objectives and design, Negligible to Low efficiency, and Negligible achievement of the project’s five original objectives, the outcome based on the original project objectives is rated Highly Unsatisfactory.

Revised objectives: Based on Substantial relevance of objectives and design, Negligible to Low efficiency, and Negligible achievement of the project’s four revised objectives, the outcome based on the revised project objectives is rated Highly Unsatisfactory.

The overall project outcome is Highly Unsatisfactory.

The project’s objectives were relevant to country conditions and to national and Bank strategy, Both the original and revised design were unwieldy and did not draw clear linkages between the objectives and the expected activities, outputs, and outcomes. Because of lack of information on the project's activities/outputs, it is impossible to construct a results chain that plausibly attributes observed outcomes to project-financed interventions. No information is provided on trends in risk behavior among key vulnerable groups. The ICR presents very little evidence of effective use of Bank resources toward the achievement of development objectives, resulting in negligible efficiency.

a. Outcome Rating: Highly Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:


The Government, through the development of a new National Strategic Plan for the years 2011-2015, has shown a strong and high-level commitment to fighting the spread of HIV/AIDS and its effects. Regional AIDS Committees have been established in all of the country’s ten regions, and District AIDS Committees in all 138 districts. HIV/AIDS focal points have been established in the ministries, departments, and agencies that received support through the project, and regional M&E focal points are operating. The ICR (p. 25) cites interviews with civil society representatives indicating that the project strengthened the country’s civil society network, particularly for large organizations coordinating and implementing HIV programs.

However, a high risk remains that these government and non-governmental bodies lack
the capacity to implement activities. The project emphasized the Ghana AIDS Commission’s (GAC’s) role in distributing funds to implementing partners for subprojects; “its overall coordination and capacity building role was neglected” (ICR, p. 26). There is no clear delineation of responsibilities between the GAC and the Ministry of Health.

Financial risk is also high. The GAC Secretariat has had limited success in increasing and diversifying financial support for its activities. DfID ended its participation in 2008, and among other donors (Global Fund and the United States Agency for International Development), uncertainty remains for funding to coordinate the national response under the GAC. At project closing, the consequences of funding gaps were already evident, as financing for PLWHA associations and support groups was reduced and activities no longer supported.

a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:

A Quality at Entry Assessment (QEA8) organized by the Quality Assurance Group in September 2007 rated quality at entry as Moderately Unsatisfactory, citing overly optimistic risk assessment, the lack of a clear link between development objectives and performance indicators, and the lack of a clear definition of responsibility and accountability for achievement of objectives. The project was prepared and became effective before the results of the GARFUND ICR, the IEG review of the ICR, and the subsequent IEG PPAR became available, so that lessons learned through the ICR process could not be incorporated. However, some lessons learned from the GARFUND mid-term review were included in the PAD (pp. 9-10): priority should be on targeted interventions for high-risk groups; the national AIDS council should define its role around facilitation and coordination rather than implementation and control; ex-post technical as well as financial audits are necessary; linkages with the Ministry of Health must be strong; and M&E systems must be strong. These lessons were not fully reflected in project design, which continued to include many interventions aimed at the general population, had weak linkages with the Ministry of Health, and did not include significant efforts to strengthen M&E (ICR, p. 14). There was no clear strategic plan for influencing the drivers of risky behaviors and therefore producing meaningful behavior change; for example, there was no social assessment work on specific issues such as orphans or at-risk groups (ICR, p. 24). There was also no effort to design a robust M&E system, and in particular, to plan for the institutionalization of a behavioral surveillance system (ICR, p. 27). Stakeholder consultations were focused on potential suppliers of project interventions, rather than on beneficiaries, with the notable exception of PLWHA (ICR, p. 24). The Bank also did not conduct an independent institutional assessment to assess the capacity gaps of the GAC Secretariat (ICR, p. 25), resulting in unclear design and implementation of technical assistance. Finally, use of the Secretariat of the Ghana AIDS Commission (GAC) as the implementing agency, minimizing the involvement of the Ministry of Health in planned project activities, downplayed the role of the ministry that should have been key for producing several of the project’s key anticipated outcomes.

Quality-at-Entry Rating: Unsatisfactory

b. Quality of supervision:

According to the ICR (p. 16), the Bank and other development partners insisted throughout implementation on a focused set of activities targeted at preventing the spread of HIV among at-risk groups, but the GAC disagreed, and there was “little room for compromise” (ICR, p. 16). The ICR (p. 27) notes that supervision was relatively less intensive during the first two years of implementation, compared with the later years of the project. A technical support team (TST) from UNAIDS and ACTafrica conducted an in-depth review of implementation bottlenecks in mid-2008, and it made a series of recommendations for changing funding criteria and prioritizing the allocation of funds to districts and NGOs, increasing collaboration with the Ministry of Health, and strengthening M&E, but no timetable was set for putting these recommendations into action, and no restructuring of the project according to these recommendations was carried out. The ICR (p. 17) calls this a “significant missed opportunity to address many of the implementation challenges and turn the project around,” but also notes that many of the Bank’s recommendations “were beyond the capacity of the GAC to implement them” (p. 28). Subsequent supervision by the Bank limited disbursements to fund only agreed activities. Overall, there was poor communication and a high level of mistrust between the task team and the GAC Secretariat (see Section 9b below). The Bank’s decision to withhold disbursement when agreed actions were not taken or when promised monitoring reports were not received exacerbated this mistrust, undermined the effectiveness of the Bank’s advice, and did not benefit the project (ICR, p. 28). Overall, despite strong efforts, Bank support was unable to improve project performance. Very little was done to address environmental issues surrounding the project (ICR, p. 19). Issues that were raised in the environmental screening and mentioned in the PAD, including training and capacity development for health care waste management, were not addressed. Furthermore, recommendations that the project support installation of disposal and packaging infrastructure as well as storage equipment at waste generation sites were not carried out (ICR, p. 19).

Quality of Supervision Rating: Unsatisfactory

Overall Bank Performance Rating: Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The ICR (p. 28) states that high-level government support for Ghana’s response to HIV/AIDS has been consistently strong and instrumental in keeping attention focused on the epidemic at the national and regional levels. Regional ministers have played an important advocacy role. However, the project contained a legal covenant committing the Government to ensure financial sustainability of the GAC Secretariat; that funding was delivered only after a lengthy delay, leading to “substantial uncertainty among the Bank and other donors about the Government’s support to the national response” (ICR, p. 29).

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:

Throughout implementation, the Bank and other donors attempted to focus the GAC Secretariat on policy making, advocacy, coordination, M&E, and resource mobilization, but instead the Secretariat remained primarily occupied with allocating small grants and managing subproject contracts. According to the ICR (p. 15), the GAC Secretariat had inadequate capacity to coordinate the national response, as shown by delays in producing annual Plans of Work, a required Sustainability Action Plan, and monitoring reports. The Sustainability Action Plan was a covenant in the project’s Legal Document, providing a framework for transferring personnel and operational costs of the GAC to the Government of Ghana. Its delay led to disagreements between the GAC and the Bank on funding of operational costs and activities, resulting in the Government delaying responsibility for payment of salaries for the GAC Secretariat staff until 2008. The GAC Secretariat also struggled to coordinate communications with other government entities, development partners, and implementing partners. There were continual disagreements between the GAC Secretariat and the Bank on the focus of activities to be funded in the annual program of work; the GAC maintained that its responsibility was to prevent the spread of the HIV epidemic to the general population, and that geographic targeting and targeting of vulnerable groups could not receive additional priority without abandoning the national scope of its mandate. Finally, there were discrepancies between the project’s financial statements and disbursement of funds, and the technical data compiled by the GAC Secretariat on the use of funds. The GAC Secretariat’s inability to provide technical audits on the use of the funds for several years was a recurring source of tension between the Bank and the GAC, resulting in delays and postponement of withdrawal of funds. Late availability of funds in turn made timely disbursements by the GAC Secretariat to the implementing partners unattainable, slowed project implementation, and had a negative impact on services to beneficiaries (ICR, p. 16).

Implementing Agency Performance Rating: Unsatisfactory

Overall Borrower Performance Rating: Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The project’s M&E plan (PAD, pp. 13, 51-53) assigned M&E responsibility at the national (GAC), regional (Regional Coordinating Councils), and district (M&E focal points) levels, including focal points in all 138 districts who were responsible for visiting all funded programs in their districts and reporting back to the regional and central levels. There was a commitment to using a single, unified set of indicators for the national response and for all donor-financed projects. This resulted in a set of indicators that did not fully or effectively measure achievement of the project’s development objectives. Several indicators were inherently flawed. The project’s original design contained HIV prevalence among adults and among youth as a key performance indicator; that indicator was dropped at restructuring, when it was recognized that prevalence is not an appropriate measure of the course of the epidemic. The indicator “median age at first sex” was not well suited for Ghana, as median age at first sex was quite high at the beginning of the project period and was unlikely to increase significantly further (ICR, p. 21). Indicators on adults and children receiving antiretroviral therapy and pregnant women receiving PMTCT were not directly linked with project activities, but instead measured the results of activities of the Ministry of Health and other donors. Baselines were available for most indicators, but were from the 2003 Demographic and Health Survey, conducted two years before the project became effective.

b. M&E Implementation:

According to the ICR (p. 17), M&E implementation was more focused on tracking financial inputs and outputs than on achieved results and outcomes. In addition, monitoring information did not link to the types of activities that were supported in annual plans of work. There was no behavioral surveillance system in place that could track information on behaviors commonly used as proxies for the spread of HIV.

a. M&E Utilization:
Activities of implementing partners were not evaluated to assess which approaches had the greatest impact on beneficiaries, or whether they had any impact at all.

M&E Quality Rating: Negligible

11. Other Issues:

a. Safeguards:

The project was classified as category “B” for environmental screening purposes and therefore triggered the environmental assessment safeguard. The classification was based on risks associated with the handling and disposal of medical waste. The project was to make use of existing arrangements for safeguard issues under the Health Sector Support Program (US$ 57.6 million, 2003-2007), as well as the provisions of a Health Care Waste Management Plan that was developed by the Ministry of Health and implemented beginning in 2005. During project implementation, however, very little was done to address environmental issues surrounding the project (ICR, p. 19). Issues that were raised in the screening and mentioned in the PAD, including training and capacity development for health care waste management, were not addressed. Furthermore, recommendations that the project support installation of disposal and packaging infrastructure as well as storage equipment at waste generation sites were not carried out (ICR, p. 19). The ICR does not state whether there was compliance with environmental safeguards; the project team confirmed that this was the case.

b. Fiduciary Compliance:

The ICR (p. 16) states that the project’s audit reports correctly reflected the disbursements of funds to the implementing partners, and financial management was satisfactory at all times. The project team stated that, because of the decision to implement the project largely as budget support, it was not clear in the early years of the project exactly how project funds were being spent. In the last two years of the project, according to the project team, there was greater effort to implement and monitor specific interventions. The project team could not provide information on whether, when, and by whom the decision to implement this Specific Investment Loan as budget support was approved. The ICR does not state whether audits were on time and unqualified; the project team confirmed that this was the case.

The ICR does not provide information on procurement. The project team stated that a list of items to be supported for procurement was created in 2010, but that most of these items had not been procured as of February 2011. According to the project team, procurement capacity was weak, and the GAC felt that Bank procurement procedures were cumbersome (even though this was a repeater project).

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Unsatisfactory
Highly Unsatisfactory
The ICR provides very little information on the project's activities (outputs), making it impossible to attribute observed outcomes to project-financed interventions and therefore construct a complete results chain. No information is provided on trends in risk behavior among key vulnerable groups.  
Risk to Development Outcome:
High
High
 
Bank Performance:
Moderately Unsatisfactory
Unsatisfactory
Identified lessons were not well incorporated into project design. Strategic assessment and planning were lacking in several areas: M&E strengthening, assessing and influencing risky behaviors among high-risk groups, and building capacity of the implementing agency. There was no clear link between development objectives, planned project activities, and key performance indicators. Environmental safeguard issues identified during preparation were not addressed during implementation. 
Borrower Performance:
Moderately Unsatisfactory
Unsatisfactory
The GAC Secretariat focused excessively on allocating small grants and managing subproject contracts, rather than more appropriate coordination and advocacy responsibilities. It failed to produce key documents and reports in a timely manner and to adopt reasonable recommendations made by the Bank, ultimately leading to a cancellation of part of the loan. 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The first lesson is drawn by IEG. The others are drawn primarily from the ICR (p. 30), with adaptation:

Focus on the high-risk groups most likely to transmit the epidemic is essential for success in preventing the spread of HIV/AIDS, even when HIV has spread to the general population. Governments are frequently reluctant to adopt this focus, and therefore considerable effort must sometimes be expended toward achieving it, including full consideration of country conditions, attitudes, and social and political environments.

While harmonized M&E systems are important, it is equally important for a Bank-financed project to be able to track the effectiveness of its specific interventions. General surveys like the Demographic and Health Surveys often do not match the start and end dates of a project, as was the case here. M&E needs to be well established before a project starts, with scheduled collections of data on well-defined key indicators at baseline, mid-term, and completion.

Especially when implementing agency capacity is limited, it is better to work with a smaller number of larger subprojects involving umbrella civil society organizations. In this case, the GAC Secretariat attempted initially to handle over 2,000 small subprojects, diverting the Secretariat’s attention almost entirely toward managing those subprojects and away from its broader coordination and advocacy function. Transaction costs can be reduced, and efficiency enhanced, by using umbrella CSOs to engage smaller ones.

Repeater projects would greatly benefit from full end evaluations of their predecessors before beginning implementation. Here, the ICR for GARFUND was produced after this project had already begun, preventing the project from incorporating what would have been critical lessons. If a complete ICR is not possible before a repeater project begins, an abbreviated ICR should be made available to help avoid continued problems.

Open communication between the Bank and the Borrower is essential to a successful project. During implementation of this project, communication broke down to the point that the project’s objectives were compromised.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR is clear and concise. It contains an extensive and nuanced discussion of some of the more complex issues related to project implementation, such as the relative responsibility of the Bank and the GAC Secretariat for the high level of mistrust that evolved between them, and the perspectives driving the Secretariat’s reluctance to focus appropriately on high-risk groups. It is appropriately cautious in its acceptance of data from the project’s M&E system (noting, for example, that the number of reported beneficiaries reached by the project far exceeded the total population of the country, ICR p. 11). However, there are shortcomings. The ICR provides very little data on actual project activities, although this is likely due to shortcomings in the M&E system; the ICR (p. 17) states that "monitoring was not linked to types of activities that were supported in the annual work plans." The ICR's discussion of the project’s efficiency is thin, with no formal economic or financial analysis and very little discussion of qualitative factors. The ICR presents two different ratings for Risk to Development Outcome: High (p. i) and Significant (p. 26). It does not state whether audits were on time and unqualified, nor whether there was compliance with environmental safeguards. Its cost and financing data are incomplete (a breakdown of project costs by component is given only for the IDA credit), and it does not provide adequate information on procurement.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions