Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Avian Influenza Control And Human Pandemic Preparedness And Response Project


  
1. Project Data:   
ICR Review Date Posted:
08/27/2012   
Country:
Albania
PROJ ID:
P100273
Appraisal
Actual
Project Name:
Avian Influenza Control And Human Pandemic Preparedness And Response Project
Project Costs(US $M)
 6.1  4.47
L/C Number:
C4206
Loan/Credit (US $M)
 5.0  3.87
Sector Board:
Agriculture and Rural Development
Cofinancing (US $M)
 0.8  0.60
Cofinanciers:
Government of Japan PHRD
Board Approval Date
  06/27/2006
 
 
Closing Date
06/30/2009 12/31/2010
Sector(s):
General public administration sector (45%), Health (27%), Agricultural extension and research (18%), Animal production (9%), Other social services (1%)
Theme(s):
Rural services and infrastructure (29% - P) Other communicable diseases (29% - P) Pollution management and environmental health (14% - S) Health system performance (14% - S) Participation and civic engagement (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Stephen Hutton
Judyth L. Twigg Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The original project objective was to: "strengthen the country’s capacity to prevent the spread of avian influenza among poultry, to prevent the transmission of avian influenza from birds to other animals and humans, and to prepare for a potential pandemic of avian influenza transmissible between humans,." as stated on page 2 of the "Technical Annex" (in lieu of a Project Appraisal Document).

The objectives in the Financing Agreement (page 5) were substantially identical.

After restructuring, the objectives were revised to include other zoonotic diseases and other infectious disease emergencies in humans: "minimize the threat posed to humans by HPAI [Highly Pathogenic Avian Influenza ] infection and other zoonoses and to prepare for, control, and respond to influenza pandemics and other infectious disease emergencies in humans," as stated in the Project Paper (page 7) dated 17 September 2009.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 10/15/2009

c. Components:
The project had four components:
A) Public Awareness and Information Campaign (appraisal estimate $0.5m, after restructuring $0.5m, actual cost $0.1m), including i) Risk Communication and ii) Communication for Behavior Change.
The component would help to develop and implement a communication plan to inform the public about the threat from avian influenza and how transmission could be minimized. The first subcomponent would include communication with the media, while the second would focus on high risk groups such as farmers.

B) Animal Health Component (appraisal estimate $3.7m, after restructuring $3.2m, actual cost $1.3m), including i) Enhancing HPAI Prevention and Preparedness Capability; ii) Strengthening HPAI Control Programs and Outbreak Containment Programs; and iii) Strengthening Disease Control Capacity, Improving Surveillance and Diagnostic Capacity, and Applied Research.
The component would support a national program to develop short-term and long-term avian influenza prevention, containment, control, and eradication activities in animals. This would include support for improved biosafety measures in the poultry sector, the creation of a National Veterinary Epidemiological Unit (NVEU), and the establishment of a compensation fund.

C) Human Health Component (appraisal estimate $1.6m, after restructuring $2.7m, actual cost $2.7m), including i) Enhancing Coordination and Program Planning; ii) Strengthening National Public Health Surveillance System; and iii) Strengthening Health Care Response Capacity.
The component included support for updating the Ministry of Health (MOH) contingency plan, building capacity at the National Influenza Laboratory through training and equipment, and strengthening the capacity to isolate and treat infectious patients.

D) Avian Influenza Task Force Component (appraisal estimate $0.3m, after restructuring $0.4m, actual cost $0.3m), including i) Strengthening the Task Force by Establishing a Permanent Secretariat; and ii) Monitoring and Evaluation.
The component would provide coordination support for the other components.

After restructuring, Component A was restructured to address knowledge gaps in the population, Component B was restructured to finance construction of a new animal health laboratory (rather than upgrading an existing one), and Component C was restructured to support revisions to the National Influenza Pandemic Preparedness Plan.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost: Most project costs were in line with those expected after the restructuring (which reallocated funds from the animal health to the human health component). The main exceptions were for the animal health component (where the planned construction of an animal health laboratory was not carried out) and the public awareness component (which was cheaper than expected as USAID and UNICEF funds were used to pay for some communication activities).

Financing: At appraisal, the project was to be financed by a $5m IDA grant, $0.8m from a Japanese Policy and Human Resources Development trust fund, and $0.3m from the client. $2.43m had been disbursed at the time of the project restructuring, out of total disbursements of $4.47m. Roughly $1.6m was canceled, due to failure to construct the animal health laboratory.

Borrower Contribution: The borrower agreed to increase its share to $0.98m at restructuring as part of financing construction of the animal health laboratory, but the final borrower contribution was only $0.25m as the laboratory was not constructed.

Dates: The project was originally expected to close on June 30, 2009. Following delays, the project was restructured on October 15, 2009, and closure was extended to December 31, 2010.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Original objectives: An uncontained outbreak of avian influenza posed a potential risk to the people and economy of Albania. Though outbreaks among poultry in 2006 had already been contained by the time the project commenced, there was a plausible and significant risk of further outbreaks in the near future. The original objective to prevent avian influenza among poultry was highly relevant because of the prevalence of backyard poultry farms in Albania where contagion risk was high, and because effective risk reduction for avian influenza requires containment at the source. Albania's capacity to deal with a potential influenza pandemic was low. The Bank's Country Assistance Strategy at completion (CAS, 2011-2014) does not specifically prioritize avian influenza, capacity building for animal or human disease surveillance, or agriculture, though it does support improving access to primary health care. However, it is Bank policy to intervene in emergency situations such as the global avian flu crisis that triggered the project, and therefore the project was relevant to Bank strategy.
Relevance of the original objectives is rated high.

Revised objectives: After the immediate threat of avian influenza appeared to have passed, the objectives were revised. The broadening of the objectives to include other zoonoses was an effective means of sustaining the relevance of the project and in assisting Albania to meet European Union animal health requirements. But removing the explicit task of preventing the spread of avian influenza among poultry reduced the project's relevance, because preventing the spread of influenza among animals is the most effective means of preventing potential influenza transmission from animals to humans.
Relevance of the revised objectives is rated substantial.

b. Relevance of Design:
Original objectives: The structure of the components was well designed to address both animal health and human health issues, following the design of the standard Bank model for avian influenza control projects. The causal framework sensibly emphasized effective communication and support to the veterinary sector and human health sector through improved institutional capacity in surveillance, diagnosis and response. However, the design also included a number of institutional capacity building and other activities that may have been overly complicated for the emergency project instrument.
Relevance of design to the original objectives is rated substantial.

Revised objectives: The revised design did not adequately support measures that would reduce the spread of disease among animals, which is a crucial part of an effective strategy for risk reduction. The revised design reduced the emphasis on controlling influenza among poultry and cancelled surveys aimed at identifying measures to improve poultry biosecurity. Despite the expanded objectives to include other zoonoses, few activities were added to support animal product management activities in the agricultural industry that are an essential part of an effective strategy for reducing the risk to humans from these diseases.
Relevance of design to the revised objectives is rated modest.


4. Achievement of Objectives (Efficacy) :

Objective 1: Strengthen the country's capacity to prevent the spread of avian influenza among poultry (original objectives only):

The project made little progress in improving biosecurity in poultry operations, either in commercial operations or backyard farms. Though a new framework veterinary law was adopted that would improve the ability to respond to outbreaks of zoonoses, as of project closure there were still no laws on biosecurity or enforcement of specific requirements. A planned survey on biosecurity was canceled. A national veterinary epidemiological unit was established, which focuses on reporting incidents of animal diseases, including avian influenza. But no progress was made in improving monitoring of wild birds, except through increased public awareness leading to voluntary reporting by the general public.

A compensation fund for culled poultry already existed prior to the project, and was used in the 2006 outbreaks. The additional compensation fund set up by the project was not used. Early response systems remained constrained by insufficient transport.

Achievement of this objective is rated modest.

Objective 2: Preventing transmission of avian influenza from birds to other animals and to humans (original and revised objectives, though the revised objectives included transmission from birds to humans only, and not to other animals):

There were no outbreaks of avian influenza during the project, so this objective can be evaluated only to the extent that the project reduced the risk of transmission in the event of a future outbreak.

Increased awareness about avian influenza and improved behaviors would plausibly reduce the risk of infection from birds to humans. The project (along with work by UNICEF and USAID) made a substantial contribution to increasing awareness about avian influenza in Albania. Many communication materials were produced, including 120,000 printed leaflets, posters, field worker communication guides and television messages, and surveys suggested they were widely viewed. Training sessions on symptoms, transmission and prevention were held for health care workers and teachers. Surveys conducted by UNICEF showed an increase in safe handling practices for poultry including washing hands (16.4% in 2006 increased to 63.8% in 2009), cooking poultry products well (21.7% to 59%) and awareness about avoiding dead poultry and wild birds (13.4% to 50.2%). The ICR reported that improved awareness increased vaccination rates by an unspecified amount.

The project led to substantial improvements in the capacity for detection and diagnosis in humans by supporting reconstruction of the virology laboratory at the Institute of Public Health to upgrade it at biosecurity level 2, together with new equipment and staff training. The effectiveness of the facility was demonstrated during the 2009 influenza pandemic, with 100% accuracy of diagnosis tests for samples that were tested locally and then sent on to an international laboratory in Romania.

Achievement of this objective is rated substantial.

Objective 3: Prepare for a potential pandemic of avian influenza transmissible between humans (original and revised objectives), and other infectious disease emergencies in humans (revised objectives only)

A National Pandemic Preparedness Plan was developed and hospital preparedness training was carried out. An Intensive Care Unit was constructed at Tirana Hospital according to international standards, and has been used for treating patients with infectious diseases.

The response to the 2009 influenza pandemic demonstrated an adequate level of preparedness. Communications strategies were tailored to the 2009 pandemic strain, some vaccines and antivirals were acquired, and the Institute of Public Health laboratory undertook necessary tests.

The project created a stockpile of 7,000 doses of antivirus, which could be used to treat infected individuals in the event of an influenza outbreak among humans. While the stockpile may expire unused, there is ongoing value from the creation of a procurement system to rapidly secure additional doses in the future.

Achievement of this objective is rated substantial.

Objective 4: Minimize the threat posed to humans by other zoonoses (revised objectives only)

After restructuring the project intended to construct a new animal health laboratory for investigating suspected cases of animal diseases, but this did not occur as the necessary building permits were not obtained. Other than the passing of a veterinary law, the project did not contribute to improvements in animal health management.

Achievement of this objective is rated modest.

5. Efficiency:

The economic analysis at appraisal was based on benefits arising from containing influenza outbreaks that were assumed to occur, but the ongoing outbreaks were contained before the project commenced and no further outbreaks occurred during the project period. The ICR correctly noted that a re-estimated ex post calculation using this methodology would not be relevant.

Traditional ex-post cost-benefit analysis is arguably not suitable for projects of this nature, where most of the benefits come from risk reduction for influenza due to improved capacity, because the probabilities of outbreaks and the impact of project activities on those probabilities are not known.

Some evidence suggests weaknesses in efficiency in design, use of resources, and timeliness. The project design included a compensation fund that was not needed, since a fund already existed, and so these project resources were unused until they were reallocated to other purposes. There were major delays in implementation, and only 11% of IDA funds had been disbursed by midway through implementation. A year into implementation the project was still hiring financial management staff who should have been in place at the outset, and the national veterinary epidemiological unit was slow to be utilized. Procurement delays slowed training activities and led to some activities not being undertaken. The animal health laboratory added in the restructuring was not built, and $1.6 million was cancelled.

Consequently, efficiency of the project is rated modest.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
44%
100%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

While the project initially had high relevance, this declined after restructuring as a focus on containing diseases at the source was removed. Though a decision was made to broaden the project to consider other zoonoses, few activities were added to achieve this objective; relevance of design was therefore substantial under the original objectives, but modest under the revised objectives. Under both the original and revised objectives, there was substantial achievement of two objectives, and modest achievement of the other. The project made substantial progress in reducing the risk of transmission from animals to humans and preparing for a potential pandemic, but made only modest progress on improving capacity to prevent the spread of avian influenza and other zoonoses among animals. While there may be many benefits from the project, inefficient design, implementation delays and the failure to construct the animal health laboratory meant that efficiency was modest.

These lead to the outcome being rated moderately satisfactory under both the original and revised objectives, resulting in an overall outcome rating of moderately satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

Coordination efforts under ministry of health and UNICEF mandates are likely to lead to continued functioning of the public awareness campaign. Risks to the animal health achievements are significant due to deficiencies in the national veterinary epidemiology unit. The ICR notes the presence of legal/administrative vacuums, internal institutional reshuffling and budgetary restrictions. Tight budgetary conditions mean that increases in laboratory capacity may be difficult to sustain, and the availability of resources to maintain the Intensive Care Unit remains unclear. The ICR did not provide information about the extent to which the capacity-building preparedness activities would be sustained through other government or donor programs after the close of the operation.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:
The Bank responded rapidly to client demand, moving from concept to approval in under 3 months. The Bank team took advantage of good practices defined under the global program on avian influenza (such as the general design covering public awareness, animal health risk reduction, and human health response capacity) and applied them to the specific country scenario. Lessons from the 2006 outbreak, such as the importance of rapid compensation payouts, particular gaps in the public outreach program, and the need to address bottlenecks in surveillance and diagnosis, were appropriately incorporated. The design was well coordinated with UN agencies and international technical organizations.

However, the design may have over-emphasized a focus on primary health care workers, given low utilization of primary health care in Albania. Many longer term capacity building objectives were insufficiently prepared due to the time constraints imposed by the emergency response approach, and may have not been suitable for inclusion in an emergency project. More clearly defined lines of responsibility in implementing agencies would have made implementation easier. The M&E design contained a number of weaknesses that contributed to poor implementation and utilization, including weak linkage between indicators and objectives and a lack of baseline information (see section 10).

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:
The Bank successfully coordinated with other partners and provided intensive supervision to deal with a number of implementation issues. Greater attention, however, should have been given to M&E implementation and utilization. Implementation and Supervision Reports reported on only a fraction of the established indicators, and reported information was not very useful for project management. Given weaknesses in the animal health component implementation in early years, the Bank should have reassessed the feasibility of these activities. The Bank could have done more to attain approval for the permit for the animal health laboratory.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The government played an important role in designing the project and in setting up a task force to manage the multisectoral (health and agriculture) project. Some confusion in reporting lines delayed implementation early in the project. The government could have done more to resolve the issue of the outstanding permit for the animal health laboratory. While the government agreed in restructuring to increase its financing contribution for the animal health laboratory, the failure to acquire permits meant that the revised government financing contribution was not met

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
Implementation was shared between the Influenza Pandemic Committee Task Force, the Ministry of Agriculture and the Ministry of Health. The task force was generally successful in a coordinating role, but there was a lack of inter-ministerial coordination early in the project and poor involvement by the Chief Veterinary Officer. A lack of experience in the Task Force with World Bank procurement and financial management procedures led to implementation delays. The Ministry of Health played a valuable role in public awareness, particularly during the 2009 influenza pandemic.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
Several weaknesses existed in the design of the M&E system. There was a disconnect between the objectives aimed at increasing capacity and the indicators that tracked actions taken. Many indicators lacked a baseline level. There were no indicators that tracked changes in the level of biosecurity in poultry management. As noted in the ICR, there were too many intermediate outcome indicators, making the monitoring system very complex. Though the M&E plan did include a rough timeline for achieving results, it did not clearly assign responsibility for data collection and analysis.

b. M&E Implementation:
The indicators were revised when the project was restructured, and some indicators were dropped. Surveys carried out by UNICEF and funded by USAID in 2006 and 2009 were useful in providing evidence on the impact of the project. The ICR reports on only 14 of the 28 intermediate outcome indicators.

a. M&E Utilization:
Gaps in staffing for M&E during the projected undermined the ability to use the project indicators to improve project management. There was little opportunity for utilizing M&E during the project because many indicators changed substantially following project restructuring.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
The project was classified as Environmental Category B, due to investments in carcass disposal for poultry and investments in laboratories and hospitals. No other safeguards were triggered. The Environmental Assessment and Environmental Management Plan were completed in July 2006, before project effectiveness. A legal review was carried out to ensure that the land where the animal health laboratory was constructed was owned by the Ministry of Agriculture and that there were no legal claims. The ICR noted that safeguard compliance was satisfactory (page 8).

b. Fiduciary Compliance:
There were a number of procurement challenges due to limited procurement management capacity that led to project delays. The Bank's requirement that procurement staff have prior experience with the Bank may have contributed to the delays, as did the decision to allow the project to utilize the project management unit of the ongoing Water Resource Management Project, which was quickly overwhelmed. Procurement reviews found that the overall procurement function was discharged satisfactorily. Financial management arrangements were assessed in December 2010 and were found to be adequate. Audit reports were delivered on time and were unqualified.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
The relevance of project objectives and design declined after restructuring. The achievement of some objectives was modest, and the project suffered from implementation delays and cancellations that adversely impacted efficiency. 
Risk to Development Outcome:
Moderate
Significant
Weaknesses in the national veterinary epidemiological unit and tight budgetary resources make sustainability of outcomes uncertain. 
Bank Performance:
Satisfactory
Moderately Satisfactory
Insufficient preparation of long term capacity building objectives and weaknesses in M&E monitoring and design weakened project implementation. 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The ICR (page 19) notes that:
  • A global template for projects can enable rapid and effective project design. The standard design for the Global Program on Avian Influenza was in place prior to this project and was used as the basis for project design, providing the structure of the components and laying out the logic for the need for containment of outbreaks among animals.
  • Projects formulated for a specific disease can be too narrow and may require restructuring. This project's original focus on avian influenza alone meant that restructuring was needed after no further avian influenza outbreaks occurred.
  • Too many indicators for an emergency project can make M&E cumbersome. The large number of indicators for this project overwhelmed its limited M&E capacity and contributed to the project's inability to implement and utilize the M&E plan.
  • Cooperation with UN agencies can enhance the quality of technical assistance. WHO, UNICEF, and USAID provided technical expertise and support that was not otherwise available from the Bank's project team or implementing agency.

14. Assessment Recommended?

Yes
Why?
As input into comparative review of the performance of avian influenza projects.

15. Comments on Quality of ICR:

The ICR is rated satisfactory, though with some shortcomings. The ICR was frank in acknowledging a wide set of procurement, staffing, and implementation problems that led to delays, and in noting poor utilization of the M&E system.. However, the efficiency section focuses almost entirely on costs and benefits rather than on cost-effectiveness or implementation efficiency, despite acknowledging that the cost-benefit analysis as designed at appraisal is irrelevant. The ICR does not provide information on the full set of indicators.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions