Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Amapá Sustainable Communities


  
1. Project Data:   
ICR Review Date Posted:
11/15/2012   
Country:
Brazil
PROJ ID:
P076924
Appraisal
Actual
Project Name:
Amapá Sustainable Communities
Project Costs(US $M)
 6.77  3.39
L/C Number:
L7265
Loan/Credit (US $M)
 4.80  2.48
Sector Board:
Environment
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  12/07/2004
 
 
Closing Date
06/30/2009 12/31/2010
Sector(s):
Micro- and SME finance (40%), General agriculture fishing and forestry sector (25%), General water sanitation and flood protection sector (25%), Other social services (10%)
Theme(s):
Other environment and natural resources management (23% - P) Rural services and infrastructure (22% - P) Access to urban services and housing (22% - P) Small and medium enterprise support (22% - P) Participation and civic engagement (11% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
John R. Eriksson
Robert Mark Lacey Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:

According to the Loan Agreement (p. 17), “the objective of the Project is to learn lessons about Amazon-specific approaches to reduce urban and rural poverty through measures that are environmentally sustainable, economically efficient and socially equitable.” Specific objectives include generating knowledge about how to: (a) reduce rural poverty through better access to social services, basic infrastructure, environmentally-sound productive activities, and credit; (b) reduce urban poverty by focusing on economic and social development in lowincome neighborhoods; (c) integrate socioeconomic development and environmental conservation at the local level through Municipal Development Fora; (d) create social capital through strengthened community organizations, capacity-building, and stakeholder participation in decision-making and responsibility for managing public resources; and (e) develop citizenship through training, participation of minority groups, local empowerment, and environmentally-sensitive decision-making.

    The PAD (p. 3) states the “goal” of the project as follows: “to learn lessons about Amazon-specific approaches to reduce urban and rural poverty. through measures that are environmentally sustainable, economically efficient and socially equitable.” It lists the same specific objectives as the Lending Agreement.
      This Review uses the statement of objectives in the Loan Agreement, since it clearly labels the objective as such, not as a “goal.” The specific objectives (a) through (e) highlight the means to achieving the learning objective.

      b. Were the project objectives/key associated outcome targets revised during implementation?
      No

      c. Components:

      1. Urban Community Development Sub-projects (US$1.79 million; actual US$0.38 million)
      Assist preparation and implementation of community development plans in poorest neighborhoods in the cities of Macapá, Santana and Laranjal do Jarí. Plans were to be prepared by each neighborhood (bairro) association in a participatory manner and encompass social development and environmental sub-projects. Anticipated sub-projects would address problems related to low levels of community organization, lack of skills and educational opportunities, absence of community planning, occupation of environmentally sensitive lands, poor sanitary conditions, and inadequate access to infrastructure and services.

        2. Institutional Strengthening for Poverty Reduction (US$1.02 million; actual US$0.14 million)
        Build capacity of the state to address key issues of poverty and development by supporting reform of the state micro-credit provider and general reform of the state administrative apparatus related to areas of the project. The microcredit subcomponent was to involve the State Development Bank (AFAP) and its Amapá Peoples' Solidarity Credit Program (AMASOL). AMASOL gave support to small urban entrepreneurs with loans between R$50 and R$5000. The administrative subcomponent was to involve the Development Agency of Amapá State (ADAP), Governor's Office and other state agencies.

        3. Rural Community Development Sub-projects (US$2.55 million; actual US$1.61 million)
        Build capacity in rural areas and provide grants for sub-projects to be identified, developed and managed by community and producer associations. Proposals to be prioritized by appropriate Municipal Development Forums to ensure consistency with the local vision for sustainable development outlined by the Forum. The component would reserve 10% of sub-project financial resources for a special set of beneficiaries: indigenous people, Afro-Brazilians (quilombolas), and poor women and adolescents. Culturally-appropriate procedures were formulated for sub-project design and approval for indigenous groups and quilombolas. The Project Coordination Unit (PCU) was to undertake a variety of measures, identified in the Special Groups Inclusion Strategy, to promote greater involvement of poor women and youths as sub-project beneficiaries.

        4. Capacity Building for Participatory Management and Technical Assistance (US$ 1.05 million; actual US$1.26 million)
        The purposes of this component were to:
        · reinforce the operational capacity of ADAP to coordinate project implementation;
        · develop a project management information system;
        · maintain a monitoring and evaluation system for the project, including close collaboration with the state Environment Secretariat (SEMA) and the national environmental protection agency (IBAMA); and
        · undertake information and promotional campaigns for public awareness.

        d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
        Cost, Financing and Instrument. The project was financed by an IBRD Sector Investment Loan (SIL) of US$4.80 million and a planned borrower contribution of US$1.97 million. Following a decision of the Midterm Review (MTR), the microcredit component was canceled on September 29, 2009, resulting in a cancelation of US$228,000 from the SIL. By project closure at the end of December 2010, US$2.48 million had been disbursed from the SIL and $0.91 million had been expended by the Borrower. Thus, a total project cost estimated at appraisal at US$6.77 million declined by closing to an actual project cost of US$3.39 million. (These figures are taken from the Financing Table in Annex 1 of the ICR; the Data Sheet on p.i shows a disbursed amount of US$2.9 million from the SIL.) The shares of the Bank and the Borrower remained almost the same from appraisal to closing, with the Borrower's share dropping from 29% to 27%. The Borrower and beneficiary was the State of Amapá, Brazil, and the guarantor was the Federative Republic of Brazil.

          Dates. The project Concept Review was held on 09/11/2002, Appraisal was on 05/03/2003, Board Approval was on 12/07/2004, and Effectiveness was 06/20/2006, 18 months after Board Approval and 45 months after Concept Review. An 18-month extension of project closing was granted, from the original date of 06/30/2009 to 12/31/2010. This was granted, according to the ICR (p.20), only after the Bank received evidence of measures taken by the Borrower to accelerate implementation. The extension occurred following the MTR, which addressed aspects of design and implementation that needed changes, in particular, adjustments to the M&E system.


          3. Relevance of Objectives & Design:

          a. Relevance of Objectives:

          The current Country Partnership Strategy 2008-2011 (CPS) mentions investments in frontier areas of the Amazon with large concentrations of poor people as an area of increased Bank action for a “more equitable Brazil.” However, the CPS also states that the Bank will provide less support for isolated demonstration projects in the Amazon. The ICR therefore concludes that lessons learned under the Amapá Sustainable Communities Project (ASCP) may in the future be less relevant for the Bank since they derived from the implementation of small, isolated rural and urban sub-projects not inserted into a larger strategic or programmatic context (p.13).

            The relevance of the project PDO to previous Government priorities was high in view of the fact that 42% of the population was considered to be poor in 2000 (compared with 45% for the Amazon Region as a whole) and 18% were considered to be extremely poor (ICR, p.1). The PAD describes two programmatic development strategies of successive State Governments of Amapá, one through 2002 and the other of a new Government, which took office in 2003. The previous Government, still in power at the time of the project Concept Review in April 2002, pursued a Sustainable Development Program based on six lines of action that closely paralleled the specific sub-objectives of the ASCP. The PAD notes that it was the intention of the Government at that stage to apply the lessons learned from ASCP through a second project (p.3). However, the post-2002 State Government formulated four strategic lines of action that were related to ASCP but not as closely as the strategy of the previous state government.
              Thus, the relevance at the time of project closing of the PDO to Bank and Borrower priorities and strategy is rated modest.

              b. Relevance of Design:
              Negligible.

                The project was first conceived as a LIL (Learning and Innovation Loan) and the intention of the State Government at the time was to apply the lessons generated by the LIL on a larger scale through a second project. But the new State Governor in 2003 wanted to follow this project with a larger loan focused on urban poverty. The Bank then decided to plan for a SIL. This decision was based on three arguments: (a) pilot projects aimed at learning lessons for a scaled-up project are typically supported through SILs, (b) the project already incorporated lessons from Bank-supported poverty programs in the Northeast and was thus not an innovation for Brazil, and (c) the processing requirements for a LIL could not be met under the circumstances, given the time that had already elapsed since Concept Review and the planned date for Approval. These reasons are not entirely consistent with each other. In particular, the last reason suggests that had the processing requirements been met, the chosen instrument would have been a LIL. In the event, the project development objective was not modified, and was explicitly framed in learning terms.

                Project design suffered from poor linkages between the overall PDO and specific objectives on the one hand, and the design of the project components, on the other hand. In spite of learning lessons being the core of the PDO, the means by which lessons were to be learned about the piloted interventions in order to contribute to the learning core of the PDO were not indicated. The most important component of a project with a learning objective is one that supports a strong evaluation process. Component design should include indications of how this core activity would be designed and implemented. No such component existed in this project.


                4. Achievement of Objectives (Efficacy) :


                1. To learn lessons about Amazon-specific approaches to reduce urban poverty through measures that are environmentally sustainable, economically efficient and socially equitable. Negligible.
                Outputs
                137 urban sub-projects were submitted for financing. Of these, 25 were completed, compared to a target of 30, of which 15 were social and 10 productive. 13 of the 25 schemes were still operating after project closure.
                24 formal urban neighborhood associations participated in the planning and choice of sub-projects. There was no target and no baseline indicator.
                18 women, youth, indigenous peoples or quilombola groups had sub-projects which were approved and funded. The PAD set a target of "10% of sub-project financial resources" to be allocated to these special groups. The ICR reports that 58% of all subprojects were for special groups, but does not report the share of financial resources they received or how many of the funded sub-projects of these groups were viable or operating at the time of project closure.
                No community development plans were prepared (ICR, p.14). The project team subsequently reported that 41 of the 52 subprojects were prepared by local communities according to their participatory development plans (as initially proposed in the PAD).
                Only one sub-project had specific environmentally-oriented activities. There was no target and no baseline indicator.
                The microcredit component was cancelled, and hence there were no related outputs.
                The planned management system was never established.
                  2. To learn lessons about Amazon-specific approaches to reduce rural poverty through measures that are environmentally sustainable, economically efficient and socially equitable. Negligible
                  Outputs
                  110 rural sub-projects were submitted for financing. Of these, 27 were completed, compared to a target of 30, of which 7 were socio-environmental and 20 productive. 13 of the 27 schemes were still operating after project closure.
                  No information is provided on the number of formal community associations or on the percentage of women in municipal development fora and community associations.
                  12 women, youth, indigenous peoples or quilombola groups had sub-projects which were approved and funded. The PAD set a target of "10% of sub-project financial resources" to be allocated to these special groups. The ICR reports that 58% of all subprojects were for special groups, but does not report the share of financial resources they received or how many of the funded sub-projects of these groups were viable or operating at the time of project closure.
                  10 sub-projects had specific environmentally-oriented activities. There was no target and no baseline indicator.
                  The proportion of investments considered financially sustainable was 43%, against a target of 50%.
                  No community development plans were prepared (ICR, p. 14). The project team subsequently reported that 41 of the 52 subprojects were prepared by local communities according to their participatory development plans (as initially proposed in the PAD).
                  The microcredit component was cancelled, and hence there were no related outputs.
                  The planned management system was never established.

                  Outcomes
                  The impact of the project interventions on urban and rural poverty was not evaluated, so that no basis was provided for a possible scaling up of the activities. There is no evidence that the project generated additional knowledge of how to reduce poverty, integrate socio-economic development with environmental concerns, or build social capital and develop citizenship through strengthened community organizations. According to the ICR (p.12), "the indicators [in the M&E system] pertaining to ... learning did not capture… the learning impact as such, but were related to the effectiveness of planning instruments, models of CFP [Calls for Proposals], institutional partnership, community capacity to formulate proposals and identification of environmental impacts.”

                  5. Efficiency:

                  No Economic Rate of Return (ERR) or Financial Rate of Return (FRR) was estimated at appraisal. The ICR provides some analysis of efficiency that focuses on differences between appraised and actual expenditures by component. It points out that less than 10% of the shortfall between total appraised costs and actual costs is explained by the cancelation of the microcredit subcomponent. There are larger differences, particularly for Components B underspending (institutional reforms related to microcredit as well as state administration) as compared with Component D overspending (project management and administration, M&E, and publicity campaign). The ICR states that: “Very little of Component B was executed, but capacity building under Component D consumed 20% more than expected at appraisal, or 37% of total project expenditure, suggesting low project efficiency" (p.16).

                  Over 59% of the loan for this operation was disbursed with mixed results in terms of outputs, and nothing to show in terms of learning outcomes. Cost-effectiveness and efficiency are therefore rated as negligible.

                  a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


                  Rate Available?
                  Point Value
                  Coverage/Scope*
                  Appraisal:
                  No
                  %
                  %
                  ICR estimate:
                  No
                  %
                  %

                  * Refers to percent of total project cost for which ERR/FRR was calculated

                  6. Outcome:

                  The relevance of the objectives was modest, and that of design negligible since the project activities included no evaluative tool to assess the project’s learning objective. There was mixed achievement of planned outputs, and no achievement of the learning objective PDO; hence, efficacy is negligible. Efficiency is also rated negligible, reflecting the operation’s low cost-effectiveness. Outcome is assessed as highly unsatisfactory

                  a. Outcome Rating: Highly Unsatisfactory

                  7. Rationale for Risk to Development Outcome Rating:

                  The new Government of the State of Amapá has decided that the project agency ADAP would not implement further community-driven development projects. According to the ICR (p.19), it is, therefore, unclear how or whether the new State government will continue to provide assistance to ongoing community subprojects such as those initiated by the project. Therefore, the risk that it will not be possible to implement a learning process based on the experience of this operation is high.

                  a. Risk to Development Outcome Rating: High

                  8. Assessment of Bank Performance:

                  a. Quality at entry:
                  A basic shortcoming of Quality at Entry was the absence of an evaluation instrument for a project with a learning objective.

                  The Bank did dedicate substantial efforts, particularly to social and environmental matters during project preparation and appraisal. However, the ICR makes a valid point that project design should have been more flexible with fewer components, to be more compatible with a small pilot project aimed at generating lessons learned. In particular, the Project Design Summary should have been shorter with fewer outcomes, objectives, results and indicators; and all of them should have had clear measurable targets (pp.19-20). The ICR also argues that project design could have been adjusted to bring about greater consistency with the PDO during the preparation period.

                  The Bank task team changed in 2003 (during preparation) without an overlap with the previous team and lacking some of its key members. Some institutional memory and continuity of expertise and experience with other relevant rural poverty programs in Brazil was thereby lost.

                  During the more than three years between appraisal and project effectiveness, several institutional changes occurred with implications for the project. These were largely the result of the change in State government in 2003. In particular, the institutional crisis in the financial institution that was to be responsible for administering the project microcredit component, AFAP, surfaced six months before negotiations and could have been addressed at that time. The ICR (p. 20) argues that a comprehensive review of project design should have been held before effectiveness in mid-2006.

                  Quality-at-Entry Rating: Unsatisfactory

                  b. Quality of supervision:

                  Supervision throughout the project appears to have been thorough. Eleven full supervision missions were carried out between late 2006 and late 2010, or about three per year. One additional mission took place out in early 2011, after project closing. In addition, Bank staff provided assistance on administrative matters to PCU staff when the latter made their annual visits to the Bank's office in Brasilia. Bank staff accompanied implementing agency (IA) staff in their review and selection of proposals and ensured that the third Call for Proposals (CFPs) benefitted from the lessons of previous CFPs. They also made field visits to about a third of the subprojects (p.21).

                  Financial management and procurement received adequate attention, including flow of funds, accounting, reporting, procurement, inventories, etc, by field office staff, and there were several special supervision missions. Collusion among suppliers and lax control over vehicle use and maintenance and over fuel consumption were detected, leading the Bank to declare several expenditure items as ineligible. Bank staff identified inconsistencies in the MIS and worked closely with the PCU to resolve the situation. The Bank reacted forcefully by going directly to the Governor when ADAP (the Borrower´s Planning Secretariat) did not make counterpart funds available in a timely manner. It also reacted strongly when the legal basis for the PCU within ADAP was struck down by a legal act of the Governor, which led to a reversal of the action (ICR, p.20).

                  The Mid-Term Review MTR addressed the essential aspects of design and implementation that needed change. An extension of the closing date was granted, but only after the Bank had received evidence of measures taken by the Borrower to accelerate implementation. An important outcome of the MTR was to clarify and simplify the M&E system. The Bank team continuously reminded the project staff of the central objective of the project - learning lessons on how to reduce poverty. But it could have worked more closely with the PCU to ensure full implementation of an M&E system that would have promoted learning. Notwithstanding these positive aspects of supervision performance, the lack of attention to the learning dimension, given its central role as the core of the PDO, brings down the supervision rating.

                  Quality of Supervision Rating: Moderately Satisfactory

                  Overall Bank Performance Rating: Unsatisfactory

                  9. Assessment of Borrower Performance:

                  a. Government Performance:
                  While the ICR observes that the Borrower (Government of Amapá) showed good ownership and
                  commitment to the project at Board approval, it had less interest in "achievement of the learning objective" (p.21). The Borrower took 18 months after Board approval to comply with conditions for loan effectiveness. During project implementation, the Borrower changed the PCU coordinator three times, along with several team members. It also changed the structure of ADAP and redistributed the PCU staff among different ADAP departments. A new state legislature and Governor dissolved the PCU. Once the Bank made it clear that this was a violation of the Loan Agreement, the PCU was recreated. Loan and counterpart funds were not made available in a timely manner to the project and its subprojects, which severely impacted project implementation and M&E during the last year of implementation. Changes in government policies and priorities in 2007-08, related to the implementation of the federal Growth Acceleration Program (PAC) led to greater emphasis on non-reimbursable sources of funds for development and decreased attention to the project and the loan from within ADAP, with significant negative impacts on project implementation.

                  AFAP never recovered from its 2004 institutional crisis, which led to the cancelation of the microcredit subcomponent of the project. The broader 2010 institutional crisis which brought charges against the then Governor and Vice-Governor brought the State administration to a halt.

                  Government Performance Rating: Unsatisfactory

                  b. Implementing Agency Performance:
                  The capacity and effectiveness of ADAP has weakened since 2003. Until 2007, it continued to exist principally to oversee implementation of the ASCP and a planned IDB-supported urban infrastructure project (which was never signed). Once loan funds became available, ADAP gave reasonable autonomy and resources to the PCU. Project staff from the PCU held at least one mission per year to the Bank´s office in Brasilia to follow up on next steps agreed on supervision missions and to get assistance on administrative matters. In 2007, when ADAP had to attend to new, larger responsibilities (i.e., to manage the Growth Acceleration Program of the State), the autonomy and efficiency of the PCU diminished, and project activities suffered significantly. The performance of the PCU improved over the period 2009 to 2010, although there was a serious delay in contracting and funding selected subprojects from the third CFP, leaving about six months for approved subprojects to be executed. It was during this time that the Borrower did not make sufficient counterpart funds available in the project account, which led to a reduction of the number of subprojects to be funded (from 15 to 9) and to delays in sub-grant disbursement.

                  Implementing Agency Performance Rating: Moderately Unsatisfactory

                  Overall Borrower Performance Rating: Unsatisfactory

                  10. M&E Design, Implementation, & Utilization:

                  a. M&E Design:
                  The Project Design Summary (PAD, p. 33) presents an M&E framework that is not consistent with the project development objective. No attention was paid to designing an evaluation system that would have permitted a rigorous identification of the lessons to be learned from the piloted interventions in order to contribute to learning what affects rural and urban poverty and how such poverty might be reduced. Moreover, there were no baseline measures of poverty in urban and rural areas. Most of the indicators were output-oriented and several lacked target values.

                  b. M&E Implementation:
                  Three complex questionnaires were designed to collect information in the field. The PCU team was overwhelmed by the task of collecting these data from about 60 subprojects on a routine basis, and attempted to have the questionnaires filled out in separate field visits from PCU supervision staff, without much success. M&E staff replaced during the first years was not trained for this task. Repeated complaints about the unwieldiness and the limited usefulness of the data to be collected led to the decision to employ a consultant in 2008 to review and revise the M&E plan. A new, improved system was adopted at the MTR, which was subsequently used more routinely. The new scheme was more conducive for producing lessons learned. It had 17 impact-related indicators, each with several “key questions” in four main areas: (i) learning lessons on how to reduce rural and urban poverty in sustainable ways, (ii) contribution to poverty alleviation, (iii) implementation of projects that add value to local production, generating income and employment, and (iv) strengthening social capital of beneficiary organizations. However, the ICR finds that the indicators pertaining to the first area did not capture learning impact as such but were related to the effectiveness of planning instruments, institutional partnership, community capacity to formulate proposals and identification of environmental impacts (p.12).

                  a. M&E Utilization:
                  The ICR does not address M&E utilization. Given the other problems that occurred during project implementation, it is not likely that many of the lessons that might have been drawn were put to use. One exception is the third CFP (Call for Proposals),which applied lessons learned from the first two CFPs. This is confirmed by the Project Team, which notes that the third CFP was prepared better, with clearer criteria, and mobilized more applicants than the previous rounds. This process has been reportedly adopted in some other States with vulnerable groups. However, as the ICR observes, "lessons could have been collected more systematically from subproject experience and data" (p.12). Most important, the data collected, as noted above, did not capture learning impact and thus failed to use information regarding the original overall PDO. The ICR also notes that "the PCU team continued to have difficulties in conducting analyses and producing reports based on monitoring data collected, in part because the system software did not allow for entering the new data and analyzing them" (p.12).

                  Given that the project objective is couched in learning terms, the relevance and quality of M&E design, implementation, and utilization are minimal. M&E quality is rated negligible.

                  M&E Quality Rating: Negligible

                  11. Other Issues:

                  a. Safeguards:
                  Three Bank Safeguard Policies were triggered: Environmental Assessment, Natural Habitats, and Indigenous Peoples. As a Category B project, a negative environmental list of activities and commodities not to be financed was provided, and specific environment-related provisions were followed during the selection process of subprojects, and during implementation and supervision. According to the ICR, subproject approval was dependent on the quality of environmental protection included in proposals as well as on the beneficiary associations having obtained necessary environmental licenses or related permissions. Safeguards related to indigenous communities were, according to the ICR, taken into account through a development plan, which included other vulnerable communities. The Special Groups Inclusion Strategy for subprojects was designed to increase representation and attention to indigenous, Afro-Brazilians (quilombolas), poor women, and adolescents. The ICR finds this strategy to have worked satisfactorily, as more than half of the subprojects funded were for the benefit of the special groups, well above the project target of ten percent (p.12). No other safeguard policies were relevant under the project. The PCU verified compliance of safeguards through field visits and checked licenses and other documentation.

                  b. Fiduciary Compliance:
                  The existence of an operational Management Information System (MIS), was an effectiveness condition, considered fulfilled by the Bank. However, some inconsistencies were found in the MIS data processing and reporting. The Bank’s FM specialist helped correct the MIS flaws and it became fully functional in 2009. But financial management and procurement issues were observed and flagged by Bank missions throughout the project. The ICR reports that during the last year, financial management and control were hampered by the fact that financial transactions could be authorized by ADAP senior management who were outside the PCU structure. Internal control mechanisms in ADAP were weak, particularly with regard to control of operating expenses and inventory of furniture and equipment. Some of these problems were solved, while others persisted. The Bank declared expenditures totaling about US$500,000 as ineligible for funding under the Loan. There is no indication in the ICR on the timeliness of project audits or whether or not project auditors’ opinions were qualified.


                    Serious procurement issues were detected during the final December 2010 mission to close the project. The issues were reported to the then Governor-Elect. A follow-up mission including an in-depth procurement post-review took place in February 2011, after the loan had closed. ADAP requested an extension of the deadline “to provide the final audit report in order to amplify the scope in view of the procurement issues” (ICR p.13). The extension was granted. The follow-up mission's findings were reported in the Bank's procurement post-review report. While the ICR does not explicitly report the conclusions, it notes that special attention was given to analysis of “procurement under the subprojects and issues were spotted in many of them, with indications of collusion, overpricing and involvement of ADAP personnel. This was reported to the new president of ADAP who immediately started an internal investigation.” (ICR, p.13).

                  c. Unintended Impacts (positive or negative):

                  d. Other:



                  12. Ratings:

                  ICR
                  IEG Review
                  Reason for Disagreement/Comments
                  Outcome:
                  Moderately Unsatisfactory
                  Highly Unsatisfactory
                  The relevance of the PDO was modest, and that of design negligible since the project activities included no evaluative tool to assess the project’s learning objective. There was mixed achievement of planned outputs, and no achievement of the learning objective PDO; hence, efficacy is negligible. Efficiency is also rated negligible, reflecting the operation’s low cost-effectiveness.. 
                  Risk to Development Outcome:
                  High
                  High
                   
                  Bank Performance:
                  Moderately Unsatisfactory
                  Unsatisfactory
                  There were major shortcomings in the Bank’s performance at entry which result in an Unsatisfactory rating for Quality at Entry (section 8a). Pursuant to the OPCS/IEG Harmonized Criteria, Overall Bank Performance is rated Unsatisfactory. 
                  Borrower Performance:
                  Moderately Unsatisfactory
                  Unsatisfactory
                  Both the ICR and IEG rated Government Performance as Unsatisfactory and Implementing Agency Performance as Moderately Unsatisfactory. Pursuant to the OPCS/IEG Harmonized Criteria, Overall Borrower Performance is rated the lower of the two (Unsatisfactory). 
                  Quality of ICR:
                   
                  Satisfactory
                   
                  NOTES:
                  - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
                  - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

                  13. Lessons:
                  The first lesson is IEG’s. The other three are taken from the ICR with some adaptation of language.

                  (1) An M&E System for Lessons Learned. For a project that has lesson learning as its PDO, an M&E system should be developed that is capable of monitoring and evaluating the effectiveness of the pilot interventions, including developing relevant indicators and an evaluation framework to measure baselines, set targets, monitor progress and measure results at the output and outcome levels, relative to a counterfactual without the intervention.

                  (2) Seek simplicity. Packaging an urban and a rural development component, microcredit, enhancement of public administration and special treatment for selected groups into one rather small loan and putting that all into the hands of an inexperienced, weak agency should be avoided.

                  (3) Timing of preparation. Long delays between appraisal and project start-up may require revisiting and revising the project before or after signing/effectiveness. When three years pass between appraisal and startup, critical elements can change -- such as key institutions, the importance and structure of the executing agency, and understanding by stakeholders of the nature and importance of the project – as can the relevance of the operation.

                  (4) Use a "Calls-for-Proposal" methodology. A CFP-type methodology of vetting local development projects can inject transparency into the process, ensure equitable chances to participate, increase respect for the autonomy of communities in deciding what they need, and, to a lesser degree, filter lower quality proposals. It can avoid favoring specific clienteles by government agencies and paternalistic or patronizing attitudes.

                  14. Assessment Recommended?

                  No

                  15. Comments on Quality of ICR:

                  While there is some repetition and not all passages are as clear as they might be, this ICR is candid and hard-hitting in its criticism of the project. It is also comprehensive and thorough. Its main deficiency is its attempt to adjust the "Project Design Summary" to simplify the summary/matrix and obtain a better fit with the original PDO and specific sub-objectives. This effort did not significantly clarify the matrix or give greater prominence to learning objectives. There is no indication of whether or not the opinion of the project auditors was qualified. There are a few inconsistencies in figures between one table and another.

                  a. Quality of ICR Rating: Satisfactory

                  (ICRR-Rev6INV-Jun-2011)
                  © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions