Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Ao-emerg Ms Recovery Erl (fy05)


  
1. Project Data:   
ICR Review Date Posted:
03/20/2014   
Country:
Angola
PROJ ID:
P083333
Appraisal
Actual
Project Name:
Ao-emerg Ms Recovery Erl (fy05)
Project Costs(US $M)
 92.00.  39.50.
L/C Number:
C4035, CH146
Loan/Credit (US $M)
 50.70  35.56
Sector Board:
Transport
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  02/17/2005
 
 
Closing Date
12/31/2007 02/28/2011
Sector(s):
Water supply (51%), General public administration sector (20%), Health (12%), Agricultural extension and research (10%), Primary education (7%)
Theme(s):
Conflict prevention and post-conflict reconstruction (33% - P) HIV/AIDS (17% - S) Rural services and infrastructure (17% - S) Education for all (17% - S) Decentralization (16% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Robert Mark Lacey
Roy Gilbert Christopher David Nelson IEGPS1

2. Project Objectives and Components:

a. Objectives:
The Development Financing Agreement (DFA, page 19) states the objectives of the project as follows: “to assist the Borrower to build a foundation for long-term reconstruction, economic rehabilitation, and the reestablishment of state administration throughout the Borrower’s territory by assisting the Borrower to: (a) improve rural incomes and enhance food security in the provinces most affected by the conflict; (b) improve access to essential education and health services in the provinces most affected by the conflict, (c) rehabilitate and reconstruct critical infrastructure; and (d) strengthen capacity of government at all levels to formulate, prepare, implement, and manage medium and long-term development programs.”

The formulation of the objective in the Memorandum and Recommendation of the President (MRP, page 8) is similar.

This Review will base its evaluation on the statement of objectives in the DFA

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
There were four components:
A. Rural development and delivery of social services (US$18 million at appraisal, actual cost not reported by the ICR). This component was to finance: (a) assistance to small farmers to increase their agricultural production by supporting the production of basic and pre-basic seeds and distribution of seeds and planting material; (b) improving the knowledge base of rural livelihood system by undertaking strategic studies, including the development of a strategy for involving producer organizations; and (c) strengthening the capacity of the Instituto de Investigação Agronómica (Institute of Agricultural Research) and of the Serviço Nacional de Sementes (National Seeds Service) to carry out activities for the improvement of the quality of seeds for multiplication schemes within the rural community.
In the social services sector, the component was to finance: (d) improved access to basic health services through: (i) increasing the number of ambulances within provincial capitals and selected areas; (ii) training of health care providers on the prevention and treatment of common diseases; (iii) promoting improved maternal and child health care; and (iv) providing essential drugs kits to health posts, health centers and municipal hospitals; (e) strengthening the capacity of public health service provision through a review of health service policies, training health care technicians and municipal health teams, and reinforcing the health information system; (f) improved quality of education services through: (i) training new and experienced teachers; (ii) providing student guides and pedagogical materials on core subjects in the provinces, and (iii) providing textbooks and other school supplies and equipment; and (g) strengthening Ministry of Education institutional capacity through improved data collection, analysis and planning, and school level management.
B. Rehabilitation and reconstruction of critical infrastructure (US$44 million at appraisal, actual cost not reported by the ICR). This component was to finance the rehabilitation and reconstruction of water supply and distribution services to the three provincial capitals of Malange (Malange Province), Kuito (Bie Province), and N’ Dalatando (Kwanza Norte Province). In Malange, it included replacement of networks (50 kilometers), 5,000 house connections and 50 community standpipes. In Kuito, it included replacement and rehabilitation of water pumps; intake stations; electrical equipment; replacement of 40 kilometers of networks; 4,000 house connections; and 40 community standpipes. In N’Dalatando, it included rehabilitation of the treatment plant at Mucari River, rehabilitation of the reservoir and construction of 16 kilometer water main to transport treated water from the plant on Mucari river.
C. Sector development strategies and strengthening of human and institutional capacity (US$7 million at appraisal, actual cost not reported by the ICR). This component was to finance strengthening the capacity of line ministries and line agencies to enable them to implement project-related activities. It included: (a) technical advisory services; (b) training programs; (c) enhancing capacity for planning and development management at the national and local levels to prepare an adequate framework for efficient decentralization and sector development strategies; and (d) carrying out a pilot decentralization in selected provinces.
D. Project management, monitoring and evaluation, and preparation of Phase 2 (US$12 million at appraisal, actual cost not reported by the ICR). This component was to finance the provision of technical advisory services and procurement of goods for (a) the management, monitoring, and evaluation of the implementation of activities carried out under the project; (b) support for activities under the National Demining Strategy Action Plan; and (c) engineering studies (including detailed design and preparation of bidding documents) and other preparatory activities for Phase 2 of the Program.

Appraisal costs reported here are taken from the MRP (Schedule A, page 25). They do not include physical contingencies of US$6 million and price contingencies of US$5 million.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost.

  • Total project cost estimated at appraisal, including contingencies, was US$92 million. The presentation of costs in the MRP is confusing, since it reports (on page 12) cost estimates for two Phases, even though the project under review consists of Phase 1 only. Little information on actual project costs, either in total or by component, is provided in the ICR. Table (a) in Annex 1 shows an estimated appraisal cost of US$51 million (which was the projected IDA contribution, not the total cost), while actual costs per component are not provided at all. There is a statement on page 18 to the effect that “at the project closing the amount of all four components was decreased to US$39.5 (IDA contribution US$34.7m).”
  • The ICR reports (page 29) that activities that were to have benefited from US$8.38 million of IDA contribution, but which could not be completed before the project’s final closing date of February 28 2011, were shifted to Phase 2. US$7.15 million of these were in the rural development sub-components of component A, US$1.2 million in infrastructure rehabilitation and reconstruction (component B), US$0.2 million in component C, and US$0.1 million in component D.
Financing.
  • According to Annex 1, table (b) of the ICR (page 28), total financing available for the project was estimated to be US$92 million equivalent at appraisal, of which US$50.7 million were committed by IDA. There are no data provided for the financing at closure.
  • Of the US$50.70 million IDA commitment, US$24.90 million was in the form of a credit and US$25.80 in the form of a post-conflict grant. Following the postponement of certain activities to Phase 2, and the decision to cancel others, a total of US$16.04 million of IDA funding committed to these activities was cancelled in July 2011, US$13.18 million from the credit and US$2.86 million from the grant.
  • Annex 1, table (b) of the ICR lists three sources of parallel financing at appraisal – the European Commission’s European Development Fund (US$7 million), France (US$1 million), and the United Nations Development Program (US$1 million).
  • France financed two high level technical assistance advisers to the Vice Minister of Planning, whom the Government had appointed as Project Coordinator. The project team explained to IEG that the first adviser, who arrived in February 2006, left the project five months later due to "lack of knowledge and the language constraint (non-Portuguese speaking)." The second served between January 2007 and December 2008. According to the project team, his contract was not renewed. The amount spent on the two technical advisers is not stated in the ICR.
  • There is little information in the ICR concerning which components the contributions from other donors were to finance, and no information on the amounts actually spent. The project team subsequently informed IEG that, except for France, no contributions materialized from other donors, and that the project was financed wholly by the Bank and the Government of Angola.
Borrower Contribution
  • A Borrower commitment of US$32 million counterpart funding was reported at appraisal, how much in kind and how much in cash was not specified. The ICR provides no information as to the contribution actually made. According to the project team, this was US$4.76 million.
Dates
  • There were three extensions to the closing date: the first, for 18 months granted on August 2, 2007, from December 31 2007 to June 30 2009, the second, also for 18 months granted on June 30, 2009, to December 31 2010, and the third granted on December 31, 2010, for two months to February 28, 2011. With regard to the last extension, the ICR reports (page 8) that the Government had requested 24 months, but IDA agreed to only two months and the postponement of certain activities to Phase 2. The extensions were requested because of extensive implementation delays resulting from an unrealistic implementation schedule, weak coordination, lack of procurement capacity, and difficult access to project sites (see Sections 8 and 9 below).


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Substantial.

  • The projects objectives are relevant to all three pillars of the Bank Group’s 2007-2009 Interim Strategy Note (ISN) for Angola (the latest available): (a) strengthening public sector management and government institutional capacity, including the reestablishment of state administration throughout the territory of Angola, (b) supporting the rebuilding of critical infrastructure and the improvement of service delivery for poverty reduction, and (c) promoting growth of non-mineral sectors. The ISN also states (page 10) that “in order to make progress towards the [Millennium Development Goals], Angola will have to substantially increase its spending on education and health services and on basic infrastructure. Due to high oil prices, government has much greater resources than appeared available a few years ago, and as a result, has significantly increased public spending on social services and infrastructure. However, the country’s development needs and capacity constraints are enormous, and policies and vehicles to underpin an effective poverty alleviation program are still being developed.”
  • The objectives are also relevant to the Government’s Poverty Reduction Strategy (Estratégia de Combate à Pobreza) and with the Priority Phase Multisector Rehabilitation and Reconstruction Program [the Government’s post-war reconstruction and rehabilitation program – PPMRRP]. Within the PPMRRP is the two-phased Emergency Multi-Sector Recovery Program, supported by the Bank as its contribution to the PPMRRP. Phase 1 of the proposed program is the project under review.

  • b. Relevance of Design:
    Modest.

  • There is no explicit Results Framework in the MRP or in either of its two Technical Annex volumes. While there is an M&E framework, it was developed only for output indicators for the project’s water sector activities (component B), for which it included arrangements for data collection and usage. A Results Framework is similarly absent from the ICR.
  • The lack of a Results Framework and assessment of the baseline situation on the ground makes it impossible to trace the causal chain from the project’s starting point and interventions and the achievement of the project’s overall development objective or even its four specific objectives. For instance, distributing seeds would likely increase farmers’ incomes and the country’s food security provided the activity were to be complemented by others, supported outside the project, but there is no indication that such activities were planned or have taken place. The only other project-funded rural development activities appear to be directed towards research and strategic studies. Similarly, access to essential education services would likely be enhanced by training teachers, supplying learning materials, and strengthening the capacity of the Ministry of Education, but only if accompanied or preceded by other activities such as reconstruction of school infrastructure destroyed in the conflict.
  • By spreading activities and resources thinly over several sectors and many activities, the project's impact was severely diluted. Such a wide range of planned outputs stretched across a large geographical area in a single project was inappropriate in a post-conflict context characterized by serious implementation and coordination capacity weaknesses.


  • 4. Achievement of Objectives (Efficacy) :

    Specific objective (a): to improve rural incomes and enhance food security in the provinces most affected by the conflict. Modest. Outputs
    • "Agricultural Development Stations" (Estações de Desenvolvimento Agrícola) in three provinces (Malange, Bie and Huambo) acquired office equipment (furniture, computers and printers) and transportation vehicles (6 trucks, 2 tractors, 16 pick-ups, and 20 motorcycles). More than 90 tons of fertilizer was produced. 200 tons of seed were produced at Huambo and delivered to farmers. 120 further tons of seed were procured and delivered to farmers. Detailed engineering and design of 685 kilometers of feeder roads in the provinces of Bie and Malange was completed.
    • Technical assistance has been provided to the Ministry of Rural Development corresponding to a first crop cycle; Planned assistance for a second crop cycle did not take place.
    • Ten National Seeds Service Technicians benefited from training in Israel. A planned training program in Brazil was not implemented.
    • Rehabilitation of the Agricultural Development Stations was started, but was not completed due to procurement-related delays.
    Outcomes
    • There is no evidence that the project succeeded in restarting agricultural production, or in multiplying seeds and planting materials. Neither is there any evidence that rural incomes increased or that food security was enhanced in the provinces most affected by the conflict.
    • The outcome indicators of the delivery of maize seeds to 20,000 families, improved cassava material to 10,000 families, and improved sweet potato material to 1,000 families were dropped.

    Specific objective (b): to improve access to essential education and health services in the provinces most affected by the conflict. Modest.
    Outputs
    Education
    • The proposed training of 1,250 teachers was not carried out because “the sector [Ministry] did not cooperate, although the consultant for that was identified and a report and training manual prepared” (ICR, page 33).
    • 348,183 books and 617,072 student kits were procured for distribution in five provinces, compared to the target of 25,000 “student guides.” The books and kits were distributed to the provinces; however, it is not stated that they were distributed to the schools and used by the children.
    • Strengthening institutional capacity through data collection, planning and management improvements was only “partially completed.” The degree of completion is unspecified.
    Health
    • Four ambulances, four auxiliary service vehicles, eight bicycles and eight motorbikes were procured and delivered to four provinces affected by the prior conflict (Malange, Kuanza-Norte, Bié and Moxico).
    • Training workshops were held on medicines management and logistics, hemotherapy, and (in part) for managers of first line health units in Moxico. “The training could not be held in Bie, Malange, and Kwanza Norte Province due to logistical constraints” (ICR, page 18).
    • Three lots of medicines for health posts, health centers, and referral units, and 6,016 essential health kits, were procured and distributed to the same four provinces.
    • The proposed strengthening of the health information system was not done because the “sector [ministry] did not show interest” (ICR, page 33).
    Outcomes
    • The ICR contains no discussion of outcomes and presents no evidence of improved access to education and health services.
    • The outcome indicator target of a 25% increase in enrollment of school children between the ages of six and twelve was dropped (reasons unspecified), and the degree of actual achievement in enrollment is unknown.

    Specific objective (c): rehabilitate and reconstruct critical infrastructure. Modest.
    Outputs
    • Rehabilitation and reconstruction of critical infrastructure in the water distribution system were completed as intended in Malange, N’Dalatando and Kuito(ICR, page 17).
    • Water standpipes in peri-urban areas were rehabilitated or constructed.
    Outcomes
    • The rehabilitation and reconstruction of the waterworks in the three cities enabled access to the distribution networks of the respective water utilities for 750 people (target 800). No further information is provided.
    • The rehabilitation or construction of water standpipes in peri-urban areas provided access to improved water delivery services to about 500,000 people compared to the target of 1.25 million.
    • The outcome indicators of producing 12-13 liters of clean water per capita per day, and reducing the cost of water from US$12.50 to US$2.50 per cubic meter were dropped.

    Specific objective (d): strengthen capacity of Government at all levels to formulate, prepare, implement, and manage medium and long term development programs. Negligible.
    Outputs
    • The planned provision of technical advisory services, the training program, and other capacity building activities did not take place “due to lack of motivation by the staff of implementing agencies” (ICR, page 8). Little explanation is provided in the ICR about the reasons for this lack of motivation. According to the Borrower’s ICR, it would appear as if the staff of the ministries concerned did not wish to devote time to project activities since they were to receive no extra payment for doing so.
    Outcomes
    · Since the proposed activities were not carried out, there were no outcomes.

    5. Efficiency:

    No economic or financial analysis of any of the project’s activities was carried out at either appraisal or completion. As this was an emergency operation (ERL), the requirement of an economic analysis at appraisal may be waived, but this exemption does not apply to completion reporting. The ICR (page 36) nevertheless incorrectly deduces that “it was not necessary at the ICR stage to recalculate NPV or ERR for the project activities completed under Phase 1.”

    There is, however, other evidence of inefficiency. It would appear that between US$35 million and US$40 million was spent for comparatively little in the way of accomplishments. A significant proportion of the activities (representing 32% of committed IDA funding) that were to have been supported by the project was not started or incomplete at closure, although the implementation period was extended by over three years. Efficiency is rated modest.

    a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


    Rate Available?
    Point Value
    Coverage/Scope*
    Appraisal:
    No
    %
    %
    ICR estimate:
    No
    %
    %

    * Refers to percent of total project cost for which ERR/FRR was calculated

    6. Outcome:

    The project’s objectives were substantially relevant to the Bank’s and the Government’s priorities for Angola. However, design relevance was modest since design did not clearly show how the activities financed, which were spread thinly over several sectors and an extensive geographical area, were expected to lead to the attainment of the project’s goals. The extent to which both the overall objective and three of the four specific goals were met is assessed as modest, since very little evidence is presented of their outcomes and hence their contribution towards attainment of the objectives. The efficacy of the fourth specific objective, related to strengthening capacity and institution-building, was negligible since most of the intended activities were not carried out. There was no estimate of the ex post economic rate of return of any sample of project activities, but the fact that a significant proportion of the planned activities (representing 32% of committed IDA funding) that were to have been supported by the project had not been started or were incomplete at closure, despite a 38 month extension of the implementation period, indicates modest efficiency in the use of project resources. With substantial relevance, and modest efficacy and efficiency, outcome is assessed as moderately unsatisfactory.

    a. Outcome Rating: Moderately Unsatisfactory

    7. Rationale for Risk to Development Outcome Rating:

    • The technical risk is significant. The MPR makes it clear (see, for example, page 9) that it was the intention to rely on long term technical assistance provided by the project to strengthen the institutional capacity of the implementing agencies, to ensure proper maintenance of the completed infrastructure in the sectors of health, education, agriculture and rural development, and to enhance environmental management capacity. However, much of the planned technical assistance did not materialize due to lack of interest on the part of the beneficiary institutions (reflecting low motivation on the part of their staff). Risks to the project-financed infrastructure from lack of adequate maintenance and management would appear to be largely unmitigated. There are also significant technical risks emanating from the incompletion of a proportion of planned activities (representing 32% of originally committed IDA funding).
    • The extent of financial risks is unknown, since there is no information provided in the ICR on the incomes and expenditures of the water utilities benefiting from the project.

      a. Risk to Development Outcome Rating: Significant

    8. Assessment of Bank Performance:

    a. Quality at entry:
    The phased approach adopted was appropriate given the country context. However, there were a number of major shortcomings in Quality at Entry:

  • Preparation under-estimated the difficulty of implementing a project with more than 100 geographically dispersed activities in a time span of less than three years in a post-conflict environment in a country with which the Bank Group had recently re-engaged after a long absence.
  • The analysis of the background information (for example, condition of existing infrastructure, soil and geological conditions) was inadequate due to limited access to the project sites.
  • Implementation and coordination arrangements were insufficient for a multi sector project involving several Ministries and implementing agencies. In an attempt to ensure adequate coordination, a Project Management and Implementation Unit (PMIU) was established under the Ministry of Planning, with the Vice Minister as Project Coordinator. However, no arrangements were made for the PMIU to be endowed with the necessary high level political backing to enable it to coordinate effectively the activities of all the implementing agencies.
  • The influence of the PMIU on the implementing agencies was also undermined by the considerably higher remuneration and superior working conditions enjoyed by the Unit’s staff compared to those of government employees elsewhere. The PMIU was staffed by local and international consultants whose salaries were paid by the project. By contrast, the motivation and response of implementing agency staff was limited (ICR, page 21).
  • Little attention was paid to reaching an agreement with the Borrower on how to address potentially serious barriers to implementation such as limited access to project sites; constrained institutional capacity of the sector implementing agencies; long delays in the procurement processing related to small activities that were more than one hundred in number; and inability of the PMIU to coordinate project activities effectively.
  • Insufficiencies in the operational readiness and capacity of private sector firms and individuals such as consultants and contractors, service providers and NGOs were not analyzed or taken into account by the project preparation team.
  • The Government had no agreed mechanism for donor coordination. The Memorandum and Recommendation of the President and its Technical Annexes contain very little discussion of how donor coordination might be enhanced. Although the Regional Operations Committee (ROC) meeting held on January 30, 2004, recommended deepening donor coordination, little action was taken.
  • Although it was intended that the project would focus only on those urgently required activities that were apparently ready for immediate implementation, it turned out that many of these activities were far from ready.
  • There were significant weaknesses in the design of the monitoring and evaluation framework (see Section 10 below).

  • Quality-at-Entry Rating: Unsatisfactory

    b. Quality of supervision:

  • Twelve supervision missions took place over the six year project implementation period. Supervision teams faced a number of significant challenges, including lengthy delays in obtaining entry visas, the difficulty in, or even impossibility of, visiting a number of project sites, the uniformly weak capacity and lack of experience of the implementing agencies, and ineffective coordination by the PMIU.
  • The Implementation and Status Reports (ISRs) produced by the supervision missions lack candor. Progress towards achievement of the development objectives is rated as Satisfactory in every ISR except the last (where it is assessed as Moderately Satisfactory). Progress in implementation is described as Satisfactory throughout except for two reports in the second semester of 2006 (which rate it Moderately Satisfactory). These assessments do not reflect (i) the cancellation or postponement of a considerable number of planned activities; (ii) that those activities that were carried out suffered significant delays, and in many cases were not completed;, or (iii) that the closing date had to be extended for a total of 38 months.
  • The mid-term review took place 18 months late. It was ineffective in addressing many of the implementation issues faced by the project. This was partly because it did not benefit from the mid-term report that had been planned. The consultant who was to assist the Project Management and Implementation Unit in producing the report had been unable to obtain an entry visa.
  • M&E design weaknesses (see Section 10 below) were not addressed during implementation.
  • The Borrower complains (Borrower’s ICR, ICR page 52) that the transition between the project’s three Task Team Leaders (TTLs) contributed to implementation delays. This is acknowledged in the ICR (page 22). According to the Borrower, after the departure of the original TTL in February 2008, “the project spent more than 12 months without receiving ‘no objection’ in procurement processes.”
  • In the circumstances, IDA’s decision not to accede to the Government’s request for a further 24 months extension beyond December 31 2010, and the concomitant cancelation or postponement of a number of planned Phase 1 activities, was appropriate.

  • Quality of Supervision Rating: Moderately Unsatisfactory

    Overall Bank Performance Rating: Unsatisfactory

    9. Assessment of Borrower Performance:

    a. Government Performance:

  • No information is provided in the ICR on the Government counterpart contribution actually made. However, the project team informed IEG that this was US$4.76 million, 15% of the appraisal estimate of US$32 million. This would appear to contradict the statement on page 10 of the ICR that the Government “has shown a strong commitment in ensuring that the counterpart funds were timely released and no payments were delayed to contractors, consultants and service providers.”
  • The Government established the Project Management and Implementation Unit (PMIU) as a condition of effectiveness. The Vice Minister of Planning was appointed as Project Coordinator, supported by a high level advisor funded by France. The Borrower’s ICR (ICR, page 52) states that the high level adviser to the Vice Minister of Planning and the internationally recruited financial management and procurement experts left the project, and that these departures affected adversely the performance of the PMIU.
  • According to the ICR (page 23), the PMIU was fully operational at effectiveness. However, later in the same paragraph it is stated that there were delays in the appointment of key national accounting, procurement and administrative staff.
  • Delays in providing entry visas significantly undermined project supervision. On several occasions, it was difficult, and sometimes impossible, to assemble the full team for supervision missions (ICR, page 21).

  • Government Performance Rating: Unsatisfactory

    b. Implementing Agency Performance:

  • The implementing agencies were the PMIU and five Ministries (those of Agriculture and Rural Development, Education, Health, Energy, and Transport). In each Ministry, a project contact manager was appointed to supervise and coordinate tasks related to project implementation. However, the capacity of the Ministries remained weak. Managers delegated tasks to their junior staff. Planned capacity strengthening activities, including training of ministerial staff, did not take place due to lack of motivation on the part of the staff concerned to become involved with the project. This lack of motivation was, in part, driven by low salaries and poor working conditions.
  • The implementing Ministries did not collect much of the technical data required for the preparation of engineering design, follow up with the procurement processing, or carry out monitoring and evaluation of project activities.
  • In the Ministry of Health, implementation was adversely affected by the withdrawal and non-replacement of the contact manager. Moreover, although provisions had been made to support local and international consultancies to provide technical advice and assist in project implementation, problems were encountered in identifying and appointing appropriate personnel. Despite this, the procurement of medicines and of transport equipment was, according to the ICR (page 23), of great benefit to the communities in the four provinces.
  • In the Ministry of Agriculture and Rural Development, the appointed contact person died, and his replacement showed little interest in the project. According to the ICR, this represented a major setback for the implementation of the project’s agricultural activities.
  • In the Ministry of Education, a consultant was engaged to assess training needs for primary school teachers and directors. However, the training could not be carried out due to lack of interest and cooperation from the Ministry’s staff. A contract was signed for carrying out engineering design and preparation of bidding documents for the rehabilitation and construction of primary schools, but on account of the poor performance of the consultant, the contract was terminated without the activity having been completed.
  • In the water sector, there were comparatively few issues with regard to implementing agency performance, and according to the ICR (page 24) the project succeeded in rehabilitating and constructing clean water systems in the three targeted provincial capital cities.
  • The PMIU was ineffective in coordinating project activities and in encouraging line Ministries to undertake their assigned tasks. This was partly due to the lack of political backing at a sufficiently high level. However, it also reflected the fact that the Unit’s staff, consisting of local consultants financed by the project, enjoyed salaries and working conditions greatly superior to those of their colleagues elsewhere in Government.

  • Implementing Agency Performance Rating: Moderately Unsatisfactory

    Overall Borrower Performance Rating: Unsatisfactory

    10. M&E Design, Implementation, & Utilization:

    a. M&E Design:
    M&E design had the following significant weaknesses:

  • The inadequate capacity of the implementing agencies to collect and analyze data for the thirteen sector-specific indicators which they were supposed to monitor was not taken into account. Also, the project did not provide for any technical support to sector agencies for M&E or for any staff training in implementing the M&E system.
  • Design of the M&E system was not carried out in consultation with the sector agencies, although they were expected to implement it (ICR, page 21).
  • There were no baseline data for most of the indicators and their target values, where they existed, were not founded on a reliable assessment of available information or of the situation on the ground.
  • The indicators were, for the most part, geared towards monitoring the outputs of the sector programs. There were few indicators for measuring outcomes, and none for assessing the degree of achievement of the project’s overall development objectives.

  • b. M&E Implementation:

  • Due to insufficient institutional capacity and difficult access to project sites, the line Ministries were unable to collect all the data necessary to monitor and evaluate progress in implementing the project. Data collection on outcome indicators was even more inadequate.
  • The number of indicators was accordingly reduced from thirteen to four outcome and one intermediate outcome indicators as follows: (i) number of poor people living in peri-urban areas with access to improved water services delivered by new or rehabilitated stand pipes and water trucks; (ii) number of people in three provincial capitals that have access to the water utility’s distribution network; (iii) tons of basic maize seed produced in the agricultural institute in Huambo; (iv) number of essential health kits distributed; and (v) rehabilitation of water utilities in three urban centers (intermediate outcome indicator). The ICR does not say when the number of indicators was reduced, and the project team was unable to supply this information.

  • a. M&E Utilization:
    The ICR reports (page 13) that the M&E data collected in relation to the five reduced indicators were used for the preparation of Phase 2.

    M&E Quality Rating: Negligible

    11. Other Issues:

    a. Safeguards:
    As an emergency operation, processed under OP 8.50, safeguards documents did not need to be approved and disclosed prior to appraisal. The project was classified as Category “B” for Environmental Assessment purposes. Other than Environmental Assessment (OP 4.01), OP 4.12 (Involuntary Resettlement), and OP 4.09 (Pest Management) were triggered. Whether the Forests (OP 4.36) and Natural Habitats (OP 4.04) safeguards might be triggered was to be determined through the environmental and social impact study (Technical Annex, Volume 1, page 35). Environment

    • The ICR (page 13) reports that “Due to the multi-sector nature of the program, and because all the environmental and social impact characteristics of the project activities were not known in sufficient detail at the time of project preparation, a programmatic process or a strategic approach for environmental and social assessment (including screening and categorization, preparation, public consultation, disclosure, review, approval and monitoring) was put in place and implemented over the life of the project.”
    • An Environmental Management Plan (EMP) was prepared. According to the ICR (page 14), “the project designed the approach to ensure that the EMP was properly implemented for all activities in compliance with World Bank safeguards policies.” All individual environmental assessments and management plans were completed within one year of effectiveness.
    • Due to local environmental and social capacity limitations, an environmental consulting firm was engaged for an initial period of three years to manage and oversee the environmental and social management of the project and preparation of Phase 2. “This firm worked closely with the Ministry of Planning and its PMIU, the ministries and agencies concerned, and the Bank to develop procedures that improved the quality and sustainability of the project activities and ensured the compliance with the World Bank Safeguard Policies” (ICR, page 14).
    • With regard to supervising environmental compliance, the project team informed IEG that the difficulties in accessing project areas were mainly at the preparation stage, and in the identification of feeder roads to be financed under Phase 2.
    Involuntary Resettlement
    • The ICR (pages 13-14) reports that “a resettlement policy framework was prepared and the Borrower followed its policy, principles, and procedures in each subproject that involved resettlement or loss of economic activities, and were disclosed separately.”
    • The same consulting firm referred to above also oversaw the management of social safeguards.
    • No further information about Involuntary Resettlement (e.g. number of households affected, land acquired, amount of compensation paid) is provided in the ICR. The project team informed IEG that "none of the project activities required resettlement," which would appear to contradict the statement quoted from the ICR two paragraphs above.
    Pest Management
    • “Because the project intended to provide pesticides, a pest and pesticides management plan was also prepared, reviewed by the Bank and implemented” (ICR, page 14).
    • No further information is provided in the ICR about pest management safeguards.
    Other Safeguards
    • There is no indication in the ICR whether or not the Forests (OP4.36) and Natural Habitats (OP 4.04) safeguard policies were in fact triggered.
    There is no statement in the ICR that Bank safeguards policies were complied with. On the basis of the information provided, it is not possible to conclude that there was such compliance.

    b. Fiduciary Compliance:
    Financial Management

    • The ICR states (page 14) that “all dated covenants in the financing agreement were complied with. The internal auditor and the independent external auditing firm were recruited. EMRP Phase 1 [the project] carried out all financial management functions including reconciliation of accounts, external audits of project accounts and the necessary actions were taken to address the weaknesses.” There is no indication in the ICR whether or not the external auditor’s opinions were qualified. The project team subsequently reported that "the Auditor expressed unqualified opinion in all reports."
    Procurement
    • The main procurement issues were the substantial delays stemming from implementing agency capacity weaknesses and the absence of systematic procedures for submitting procurement documentation from the implementing agencies to the PMIU and then on to IDA. There were no reported cases of misprocurement (ICR, page 14).

    c. Unintended Impacts (positive or negative):
    None reported.

    d. Other:



    12. Ratings:

    ICR
    IEG Review
    Reason for Disagreement/Comments
    Outcome:
    Moderately Satisfactory
    Moderately Unsatisfactory
    The extent to which both the overall objective, and three of the four specific goals, were met is assessed as modest. The efficacy of the fourth specific objective, related to strengthening capacity and institution-building, was negligible since most of the activities were not carried out. There was no estimate of the ex post economic analysis of the project, but the fact that a significant proportion of planned activities had not been started or were incomplete at closure, despite a 38 month extension of the implementation period, suggests modest efficiency in the use of project resources. With substantial relevance of objectives, modest relevance of design, and modest efficacy and efficiency, outcome is assessed as moderately unsatisfactory. 
    Risk to Development Outcome:
    Moderate
    Significant
    Risks to the project-financed infrastructure from lack of adequate maintenance and management are significant, and would appear to be largely unmitigated. Financial risks are unknown, since there is no information provided in the ICR on the incomes and expenditures of the water utilities benefiting from the project.  
    Bank Performance:
    Moderately Satisfactory
    Unsatisfactory
    Major weaknesses in Quality-at-Entry, which is rated unsatisfactory, included an over-complex design, with activities thinly spread over several sectors; an over-ambitious timetable and inadequate implementation arrangements, especially in a post-conflict situation; under-estimation of the impediments to implementation; and insufficiencies in M&E design (see Section 8 above). 
    Borrower Performance:
    Moderately Satisfactory
    Unsatisfactory
    Government counterpart funding shortfalls, policies and procedures with regard to the granting of entry visas posed a major challenge to project supervision. There were delays in key appointments to the PMIU and implementing agencies. Implementing agency performance was characterized by substantial implementation delays (and in some cases outright failure to implement) and lack of staff motivation to engage in project activities (see Section 9 above). 
    Quality of ICR:
     
    Unsatisfactory
     
    NOTES:
    - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
    - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

    13. Lessons:
    The following lessons, partly drawn from the ICR, can be gleaned from the experience of preparing and implementing this project:
  • Project design, particularly in a post-conflict context, should carefully identify and analyze barriers to implementation and include measures to address them. In this case, factors that were not taken fully into account included lack of, or difficult, access to many project sites, lack of motivation in the implementing agencies, and capacity weaknesses in the private sector and civil society.
  • In order to be effective in a post conflict context, it is critical to build enough flexibility in the project design to meet the changing needs that could not be conceived at the time of project preparation on account of limited and unreliable data availability. In this case, many of the planned activities turned out to be infeasible.
  • Projects in post-conflict countries, especially where (as in this case) the Bank Group has been absent for a lengthy period, should be kept simple, with clear, measurable objectives and uncomplicated implementation arrangements involving a minimum number of implementing agencies.
  • Arrangements based on a project financed implementation unit should ensure that the unit has sufficient political backing to carry out its coordination and management mandate, and that resentments caused by the higher salaries and superior working conditions afforded to unit staff do not undermine motivation elsewhere in the public service.
  • Terms of reference and related compensation procedures for international and local consultants should adequately reflect the goal of transferring capacity to their government counterparts. In this case, the ICR reports (page 26) that consultants did not pay sufficient attention to the training of counterparts.

  • 14. Assessment Recommended?

    Yes
    Why?
    Once the Phase 2 project is complete, a Project Performance Assessment Report of both operations together could provide useful lessons on multi-sector projects in a post-conflict environment. It would also verify the ratings.

    15. Comments on Quality of ICR:

    The main weakness of the ICR is the absence of an evidence-based assessment of project outcome. The document simply asserts that outcome was moderately satisfactory based on the delivery of certain outputs. No evidence is provided in the ICR that the activities of the project have led, or could be expected to lead, to the overall result intended. The ICR’s task in this respect was made difficult by the fact that there were few, if any, outcome indicators in the project’s Results Framework. Even in terms of outputs, however, it is difficult to reconcile the project’s reported achievements with the ICR’s outcome ratings. Equally incongruous is the dichotomy between the textual discussion of Bank and Borrower performance and the ICR's performance ratings. There are a number of further drawbacks. The project’s development objectives are not discussed in relation to the Bank’s Interim Strategy Note for Angola. The ICR erroneously affirms that no ex post economic analysis is required for emergency projects. There is no information on project costs by component at appraisal or closure (component costs at appraisal are for the IDA funding only, and there is no information on project costs by component at closure). There is no information on how much of the contributions of other donors was actually spent or on what (the European Development Fund’s contribution is not mentioned at all in the main text). The project team subsequently informed IEG that "counterpart funding contributed to approximately 30% of the project as reported by the PMIU." The size of the actual Borrower contribution is not specified. The dates when the extensions were granted are not provided. The dates of departure from the project of the foreign financed advisers are not specified (the project team later stated that they were replaced by local consultants "approximately in 2009"), nor are the reasons for their departure. The date when the indicators were reduced from 13 to five is not given according to the project team, "the number of indicators was not actually reduced, but the Bank guidelines at the time were different and not as strict as today in regards to monitoring and reporting of results. Thus, no all indicators were measured and reported in the project ISRs"). There is very little supporting analysis or information on compliance with involuntary resettlement and pest management safeguards. The project team subsequently provided some additional information (see Section 11a above). It is not stated whether or not the external auditor’s opinions were qualified (the project team subsequently reported that "the Auditor expressed unqualified opinion in all reports"). On page 9 of the ICR, it is stated that the Vice Minister of Finance was appointed as Project Coordinator, but on page 23, it is the Vice Minister of Planning (the project team later explained that a Vice Minister of Planning was the Coordinator, but that, in 2009, he was appointed as a Vice Minister of Finance, but continued to serve as Project Coordinator until May, 2010).

    a. Quality of ICR Rating: Unsatisfactory

    (ICRR-Rev6INV-Jun-2011)
    © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions