Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Second Urban Poverty Project (upp2)


  
1. Project Data:   
ICR Review Date Posted:
06/30/2013   
Country:
Indonesia
PROJ ID:
P072852
Appraisal
Actual
Project Name:
Second Urban Poverty Project (upp2)
Project Costs(US $M)
 126.69  507.35
L/C Number:
C3658, L4664, L7752
Loan/Credit (US $M)
 100.00  257.69
Sector Board:
Urban Development
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  06/13/2002
 
 
Closing Date
06/30/2008 12/31/2010
Sector(s):
Health (27%), Roads and highways (27%), General education sector (27%), Other social services (13%), Sub-national government administration (6%)
Theme(s):
Urban services and housing for the poor (23% - P) Municipal governance and institution building (22% - P) Health system performance (22% - P) Participation and civic engagement (22% - P) Poverty strategy analysis and monitoring (11% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Robert Mark Lacey
George T. K. Pitman Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The project under review is the second in a series of urban poverty projects in Indonesia. The first such project became effective in 1999 and closed in 2004. The second operation became effective in 2002 and closed in 2010. A third project was approved in 2005, and was implemented in parallel to this project, closing in 2011.

According to the Project Appraisal Document (PAD, page 2) and the Development Credit Agreement (page 13), the project development objectives of this second operation in the series are:

    " (i) to establish or support representative and accountable community organizations that are able to provide services to the urban poor and increase the voice of the poor in public decision making; (ii) to make local government more responsive to the needs of the poor through increased cooperation with community organizations; and (iii) to improve services for the urban poor (financial services, social services and infrastructure).”

In April 2007, Additional Financing in the amount of US$135.5 million was approved. The Project Paper (page 3) states that “changes to the existing project include: (a) geographic expansion to new areas; (b) adoption of a more rigorous anti-corruption action plan; (c) strengthened financial management, procurement, and disbursement arrangements; and (d) improved training program for consultants and facilitators, including modules on community-based disaster management. These revisions are consistent with the project’s objectives, which will not change, and will improve the effectiveness of project implementation.”

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
There were four components:
A. Community development and local government capacity building (US$17.08 million at appraisal, US$29.24 million at closure). This component provided technical advisory services to carry out community
development and local government capacity building activities, including: (a) the formation (or confirmation) of an elected body of representatives known as the Board of Community Trustees (hereafter Trustees); (b) the formulation of a community development plan for each project kelurahan (ward with a community organization), using a transparent and participatory process, and (c) assistance to Trustees on how to form associations (Trustee Fora)
and to local governments to build capacity for working with Trustees and Trustee Fora. The component was to involve a guided socialization process where facilitators were trained to work directly with communities in project kelurahan and with local government officials at provincial, district and municipal levels.

B. Kelurahan grants (US$58.82 million at appraisal, US$325.93 million at closure). This component was to finance block grants to participating kelurahan to finance activities identified in their community development plans (CDPs), including: (a) specific high-priority infrastructure investments, (b) competitive proposals from community groups consistent with the priorities in their CDPs, (c) microcredit loans for community groups, and (d) social safety programs for the benefit of the poorest and most vulnerable groups or individuals. Kelurahan grants would not finance activities on the project’s negative list (such as religious buildings or government offices). Each participating kelurahan would receive a one-time block grant of between IDR 150 million and IDR 500 million (US$16,600 to US$55,400) to finance investments in one of more of the above categories, in accordance with their CDPs.

C. Poverty alleviation partnership grants (US$35.23 million at appraisal, US$88.35 million at closure). The purpose of this component was to encourage partnerships between local government and communities and to institutionalize a consultative process between the partners for future activities undertaken by local governments. It was to provide local governments with access to matching grants (the poverty alleviation partnership grants) to finance poverty alleviation sub-projects that are: (a) too big to be financed by the kelurahan grants or that require local government involvement, (b) located in more than one kelurahan, (iii) not on the negative list for kelurahan grants, and (iv) jointly prepared, proposed and implemented by Community Organization (Badan Keswayadan Masyarakat) in collaboration with local government departments. Both participating local governments and eligible sub-projects were to be selected on a competitive basis.

D. Implementation support (US$9.73 million at appraisal, US$58.49 million at closure). This component was to support project management by a Project Management Unit in the Ministry of Public Works. The Unit would hire consultants and facilitators to assist in project implementation. Technical assistance would be provided through national management consultants at the central level, and oversight consultant teams at the province level. The oversight consultants were to have offices in the participating district governments, and facilitators and community cadres at the kelurahan level. National management and oversight consultants would also assist the Borrower with monitoring. Evaluation would be done separately by evaluation consultants independent from the national management consultants.

US$5.83 million at appraisal and US$5.34 million at closure were not allocated by component.

The components were not revised, but the scope, costs, and Bank financing of the activities were increased and funded by the Additional Financing.

In April 2005, the project was restructured so that it could provide emergency assistance and reconstruction grants to the urban poor affected by the 2004 tsunami and earthquake. In October 2005, an agreement was signed to channel an additional US$1.26 million through the project from the Japan Social Development Fund for this purpose. However, because the Multi-Donor Trust Fund established for disaster relief was already deploying resources through other, dedicated disaster-relief projects, these activities were not implemented under the project, and the Japanese grant was cancelled. “The restructuring was never actually implemented or integrated into the project, and the project essentially continued as though the restructuring had not taken place” (ICR, page 5).

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost.
Total project cost at closure was US$507.65 million (including an IBRD front-end fee of US$300,000), about four times the appraisal estimate of US$126.99 million. The greater part of the increase was in components B and C – the Kelurahan grants and the poverty alleviation partnership grants, although the greatest proportional increase was in component D, implementation support, which rose by over five times from US$9.73 million to US$58.49 million.
From 2006 onwards, within component B, the proportion of Kelurahan grants allocated for microcredit loans for community groups was restricted to 20%, and this was subsequently reduced to 10% from 2007 onwards. This was in response to the relatively poor performance of these loans during the first three years of implementation (ICR, page 5).

Financing.
The only external financing for the project came from IBRD and IDA. IBRD funds were provided through two Loans. The first, IBRD-46640, was approved n June 11 2002. In July 2009, this was converted into IBRD 77520. At the time of the conversion, only US$2.48 million had been disbursed. However, the remaining US$27 million, which had become Loan 77520, had all been disbursed by project closure. At the same time as the approval of the original IBRD Loan, on June 11, 2002, the Board approved an IDA Credit of US$70.5 million. The actual amount disbursed from this Credit was US$81.14 million, thanks to the appreciation of the SDR against the US dollar during the life of the project. The Additional Financing of US$135.5 million, approved on May 22, 2007, was entirely from IDA resources. Again, thanks to exchange rate appreciation, the dollar value of disbursements from the Additional Financing was US$139.63 million. Nearly all the extra dollar funds resulting from exchange rate movements were absorbed, and only US$0.3 million was cancelled at project closure. There was also a Japanese Trust Fund of US$1.26 million (TF55389, approved on January 24, 2006), which was to finance a pilot housing program for vulnerable communities . According to the Operations Portal, this Trust Fund was closed in November 2007, without any request for disbursement having been received before the Grant period had expired. There is no mention of this Trust Fund in the ICR.
Borrower contribution.
Contributions of the Government and communities were, respectively, US$140.09 million and US$109.88 million, many times the appraisal estimates of US$13.29 million from the Government and US$13.71 from the communities. This reflected in large part the expansion of the program to national scale.

Dates.

Three extensions to the closing date were granted, for a total of 30 months. The first, six month extension (from June 30, 2008 to December 31, 2008) was authorized at the signing of the Additional Financing in April 2007, because it was anticipated that the Government would need additional time to on-lend an extra US$135.5 million to a much larger number of kelurahan than originally envisaged. A second extension of twelve months to December 31, 2009, was to allow time to disburse funds to kelurahan that had not received block grants because of delays in national government budget execution and mid-year budget revisions, as well as difficulties in mobilizing and retaining facilitators (partly because of delays in salary payments). A third, and final, extension, also of twelve months, to December 31, 2010, was granted in part to enable the Government to utilize part of the project proceeds to support urban neighborhoods in West Sumatra in their recovery from the earthquake that hit the province in September, 2009. However, the extension was also required because of recurring delays in budget execution, budget revision and in the mobilization of facilitators to assist the expanded number of beneficiary communities. The ICR does not specify the dates on which the last two extensions were granted. The project team subsequently stated that the approvals were on December 19 and 28, 2008, respectively.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High.

  • Project objectives were, and remain, relevant to government policy. In 2006, the Government launched the first nationwide poverty reduction program, comprising two pillars: (a) the National Program for Community Empowerment (Program Nasional Pemberdayaan Masyarakat), and (b) the Conditional Cash Transfer Program targeting poor communities. The key objectives of the policy were to reduce poverty from 18% in 2007 to 8% in 2009, and unemployment from 10% to 5% over the same period. These efforts would be achieved through nationwide community-driven-development and labor-intensive activities. The Additional Financing provided through this project in 2007 extended Program implementation in about 7,300 kelurahans in all 33 provinces. The project was, therefore, also highly relevant to the Government’s pursuit of its ongoing strategy of community-driven-development, as initiated by the 2000-2004 National Development Plan. The Government remains firmly committed to such endeavors, and has repeatedly stated that it will continue to implement its national community-driven-development program at least until 2014.
  • The project’s development objectives are highly relevant to the World Bank Group's 2009-2012 Country Partnership Strategy (CPS) for Indonesia, current at closure. Community development and social protection is one of the five thematic areas that were expected to form the core of the Bank’s engagement. The CPS (pages 26-27) states the Bank: “will continue to support Government in the design and effective expansion of programs that promote more inclusive growth and social protection, and that also enhance the accountability of elected officials and service providers…….The CPS approach inherent in the partnership between the Government and the WBG is built around the National Community Empowerment Program (PNPM-Mandiri), which is based on the well-performing Kecamatan Development Project (KDP) and the Urban Poverty Project (UPP) models — programs in which the World Bank Group has had a long-standing engagement.”
  • The project directly addressed two of three priority areas of the Bank’s 2001-03 Country Assistance Strategy (current at appraisal): (i) sustaining economic recovery and promoting broad-based growth, and (ii) delivering better public services for the poor.
  • The continued relevance of project development objectives is affirmed by the request for Bank support for follow-on operations such as the National Program for Community Empowerment in Urban Areas.

b. Relevance of Design:
Substantial.

  • Although the PAD does not include a Results Framework as this would be currently understood, the project’s development objectives were clearly stated, and there is a logical causal chain between the activities to be supported, the outputs expected and the intended outcomes. The components were well designed, realistic, and not overly complex. They were adapted from proven approaches developed in the First Urban Poverty Project. The first objective – to establish or support representative and accountable community organizations that are able to provide services to the urban poor and increase the voice of the poor in public decision making – would be directly supported by the capacity and institution-building activities in component A and by the national and local level consultants and facilitators financed by component D. Efforts to attain the second objective -- to make local government more responsive to the needs of the poor through increased cooperation with community organizations – would be underpinned by the inclusion of the poverty alleviation partnership grants which would avoid by-passing established local governments and hence hampering the development of productive and sustainable relationships with communities. The third objective – to improve services for the urban poor – would be supported by the kelurahan and partnership grants. However, one notable design shortcoming was the absence in the PAD of targets for the outcome indicators. This was corrected in the Project Paper for the Additional Financing.


  • 4. Achievement of Objectives (Efficacy) :

    The project development objectives were (i) to establish or support representative and accountable community organizations that are able to provide services to the urban poor and increase the voice of the poor in public decision making; (ii) to make local government more responsive to the needs of the poor through increased cooperation with community organizations; and (iii) to improve services for the urban poor (financial services, social services and infrastructure).
    This project followed the First Urban Poverty Project, and was implemented in parallel with the Third Urban Poverty Project. All three operations had broadly the same aims. Attribution of results to individual projects is, however, facilitated by the fact that geographical coverage was different in each case (see ICR, page 6, PAD, page 7, and page 27 of the PAD for the Third Urban Poverty Project), although the third project did also provide some additional support for 660 Boards of Trustees already selected under the first and second operations.

    (i) To establish or support representative and accountable community organizations that are able to provide services to the urban poor and increase the voice of the poor in public decision making. Substantial.
    Outputs
    • 99% (target 100%) of all participating kelurahans completed and ratified community development plans for the provision of services to the urban poor, and 67% of all plans were implemented.
    • 99% (target 90%) of beneficiary communities, local authorities and other targeted areas received technical support from national management consultants, oversight consultant teams and other facilitators.
    • 87% (target 90%) of the oversight consultant teams provided accurate and timely data on the operations of kelurahans, and cooperation with local governments through the management information system.
    Outcomes
    • The target of at least 5,092 (or 70%) of the Boards of Community Trustees operating in a representative, effective and participatory manner was exceeded (ICR, page 20), based on (i) their composition, participation and voting rates, (ii) women’s participation, (iii) complaint handling, and (iv) their success in identifying infrastructure priorities, creating community development plans, and carrying out the required investments. The evidence for this is based on the following outcome and intermediate outcome indicators which were monitored in the project’s management information system:
        • 19% of the population (kelurahan members) participated in community discussions during project implementation (target 30%).
        • 2% of the adult population voted in the final stage of Board of Trustee elections (target 3%).
        • 44 community volunteers were recruited per kelurahan (target 25).
        • 66% of the kelurahan population was aware of the project and of its objectives (target 25%).
        • 22% of those elected to Boards of Trustees were women (target 20%).
        • The percentage of kelurahan populations aware of community development plans is not yet available (it is awaiting the results of the government survey); the target was 25%.
    • The following three indicators were added at the time of the approval of the Additional Financing:
        • 42% of women and other vulnerable groups participated (target 30%).
        • 35% of the adult population voted in Board of Trustee elections at the neighborhood level (target 30%).
        • 99% of Boards of Trustees completed annual financial audits (target 44%).
    • According to a Government survey made available after ICR completion, 73% of the Community Organizations (Badan Keswayadan Masyarakat – BKM) are regarded representative, effective, and operate in a participatory manner (the target was 70%).
    • While the community organizations met all quantitative targets of voter turnout and women representation, thereby suggesting that they were generally representative and accountable, the ICR also notes (page 18) that the average participation level of kelurahan members as a whole (poor and non-poor) was lower than expected (19% against 30%). Moreover, some of the targets (for example, 2% of the adult population voting in the final stages of BKM elections) seem less than ambitious.
    • There is no evidence concerning handling of complaints and ensuring good governance, although the ICR states on a number of occasions that this worked well throughout implementation.
    (ii) To make local government more responsive to the needs of the poor through increased cooperation with community organizations. Substantial.
    Outputs
    • 40 poverty alleviation partnership grants (PAPG) cities were selected to participate in the project (target 30).
    • 40 PAPG selection committees were formed (target 30).
    • The number PAPG sub-projects completed per city was 81 compared to a target of 40.
    • The Central Government has issued a policy paper for long term poverty alleviation.
    • 57% of participating local governments formed Trustee Fora (target 40%).
    Outcomes
    • The contribution of local government cofinancing in poverty alleviation partnership grant-supported initiatives was 43% (target 25%). This cofinancing was funded from the local governments’ own budgets, and enabled considerable leverage of the Bank’s contribution. It reflects in part changes in financing arrangements. Since 2008, provincial, district and municipal level governments chosen to participate in the project are required to finance kelurahan grants from their own budgetary resources. Since 2009, they are also required to finance at least 50% of PAPG grants from their own budgets (ICR, page 5).
    • In addition to funding investments, 117 local governments provided their own resources for project monitoring, partnership with kelurahans, channeling resources to support community development plan implementation etc. (the final target, was revised upwards after the availability of Additional Financing, was 78 local governments).
    • Three indicators – (i) the percentage of local government staff in cities aware of the poverty alleviation partnership grant program (target 30%); (ii) the number of poor people among direct beneficiaries of grant services as a percentage of the poor in grant cities (target 1.6%); and (iii) the percentage of kelurahan populations in grant cities aware of the grant program (target 30%) – are pending the results of the Government’s survey.
    (iii) To improve services for the urban poor (financial services, social services and infrastructure). Substantial.
    Outputs
    • As a result of the Additional Financing, the project was extended to new areas and coverage was expanded from 2,227 to 7,273 kelurahans.
    • About 58% of kelurahan grants were invested in physical infrastructure, primarily in roads, bridges, irrigation, drainage and water supply as shown in the following table:
      Type of infrastructureUnit measureOriginal projectAdditional FinancingTotal
      Village roadskm2,7996,9829,781
      Small bridgeskm73188261
      Irrigation canalskm228129357
      Drainagekm9832,4513,434
      Water supply facilitiesunit10,67939,22949,908
      Sanitation facilitiesunit13,29139,24852,539
      Markets and shopsunit951,4491,544
      Houses rehabilitatedunit12,97049,29062,260
      Water disposal facilitiesunit5,95733,35339,310
      Electricity connectionsunit5,13914,21919,358
      Community health facilitiesunit1,1165,1386,254
      Education facilitiesunit445,0675,111
    Source: ICR, page 18.
    • About 27% of the kelurahan grants were on-lent as micro-credit loans. The remaining 15% of the kelurahan grants financed social infrastructure and services. Grants for microcredit loans and social services were channeled to the poorest residents of the project’s kelurahans (ICR, page 11).
    • 56% (target 35%) of all poor households in kelurahan covered by the project benefited from infrastructure grants, and 53% (target 10%) of poor families received social grant assistance.
    • 61% of grant recipients (target 30%) were women.
    • The project provided 19% (target 15%) of all poor households in its coverage area with access to microcredit through revolving funds.
    Outcomes
    • According to an impact evaluation survey conducted in three rounds between 2004 and 2007, the project resulted in an improvement in access to adequate sanitation, in particular for the poor. The majority of persons who received project-funded credit, although already having access to credit, were able to obtain lower interest rates thanks to the project. However, the survey was unable to identify a statistically significant impact of the project on the welfare of the population living in project areas. The survey is not included in the ICR, nor is there any information on its methodology.
    • No similar survey could be conducted after 2007, since nearly all kelurahan were covered by the nationwide Community Empowerment Program – known as PNPM - and a control group could no longer be constructed. The results of a Government survey, based on an assessment of the project’s key performance indicators in 2011, were still awaited when the ICR was completed.
    • The project team subsequently made the results of the Government survey available to IEG. According to the survey: (i) at least 37% of revolving fund recipients have increased their income level, compared to the 2010 target of 20%; (ii) 73% of the Community Organizations (Badan Keswayadan Masyarakat – BKM) are representative, effective, and operate in a participatory manner (the target was 70%); and (iii) 75% of the kelurahan population is aware of the project and its objectives (the target was 25%).
    • According to the ICR (page 20), 92% of beneficiaries expressed satisfaction with the improved level of services provided, exceeding the target of 80%.
    • The micro-credits demonstrate only moderate levels of sustainability – the percentage of revolving funds with repayment rates in excess of 90% was 49% (target 70%). In response to the relatively poor performance of microcredit loans prior to the availability of the Additional Financing (2003-2006), the allocation of kelurahan grants to the revolving funds was restricted to no more than 10% from 2007 onwards. As a result, the share of kelurahan grants allocated to microcredit loans dropped from 36.6% in the original project to 11.7% in the Additional Financing.

    5. Efficiency:

  • Given the low cost of the sub-projects financed by the project and the substantial economic benefits of infrastructure investments in locations where infrastructure is largely underdeveloped, the ICR did not deem it useful to estimate indicators such as net present values or economic or financial rates of return.
  • Instead, efficiency was gauged by “unit rate norms” (ICR, Annex 3, page 23). These are described as “highly favorable for community-executed infrastructure projects.” Supporting evidence was drawn, not from the project under review, but rather from three other sources. First, a 2005 study by Indonesia’s National Planning Board is cited, which found that “the average construction cost of community-driven-development projects was 40% lower than those done by contractors of local governments. The cost savings were mainly attributed to the high level of voluntary labor contributed by residents in community-driven development projects.” Second, small-scale infrastructure constructed by communities in a related Bank-financed project (the Community-Based Settlement Rehabilitation and Reconstruction Project for Nanggroe Aceh Darussalam and Nias) was found to have cost 23% less than similar projects undertaken by local governments. Tendering of materials by communities reduced their costs by 12%. Third, a November 2011 review of Indonesia’s National Community Empowerment Program (Urban) by the Rand Corporation “commented favorably on the cost-effectiveness of infrastructure projects undertaken financed by UPP2 [the Second Urban Poverty Project, under review] and similar projects.
  • The study was not able to assess in detail the cost-effectiveness of UPP2-financed projects because (i) budget data was deemed insufficient or questionable, (ii) counterfactuals were not available, and (iii) the study team lacked suitable measures of cost- effectiveness (in terms of impacts)” (ICR, Annex 3, page 23). However, according to the project team, “the conclusions of these PNPM-wide surveys also apply to UPP2 because the UPP2 loan supported precisely the same type of infrastructure projects (which were developed based on the same approaches) as projects financed from other funding sources for PNPM. In fact at the time of UPP2 completion, it has been considered as part of the overall PNPM urban. The studies under PNPM [are] consistent with UPP2. Qualitative study completed after ICR confirm this assumption take by the Team for ICR.”
  • The average size of kelurahan grant per beneficiary kelurahan was US$44,813 at closure, compared to an appraisal estimate of US$26,412. Total project cost per beneficiary kelurahan was US$69,799 at closure, compared to an appraisal estimate of US$57,022. The project team subsequently stated: “The increase in kelurahan grants does not necessarily represent an increase in cost, because the project finances priority needs of community, and additional grants finances more activities. It is not cost overrun. Extra cost increases the benefits, because the grants financed much more small-scale infrastructure––the economic benefits of which are documented in Annex 3 of the ICR––than was envisaged at the time of appraisal (especially roads).”
  • Implementation support costs (component D) increased by proportionally more than the project cost as a whole. According to the project team, “higher than foreseen implementation costs also do not necessarily indicate that that the project’s efficiency was modest. The number of facilitators (and their salaries) were deliberately increased in order to improve the quality of the grant-financed project and increase the number and ensures sustainability. Stated differently, the implementation cost was increased in order to generate additional benefits and significantly more output. The net effect is this measure is not automatically a reduction in cost efficiency.” Such increases were justified by persistent problems with the recruitment and retention of facilitators. According to the ICR (page 15), “many of the implementation issues could have been avoided if more funds and attention were allocated to training of facilitators.”
  • The project’s closing date was extended for a total of 30 months. The first extension (6 months was granted at the time of approval of the Additional Financing to make allowance for the enlarged scope of the project. The second extension (12 months) was related entirely to inefficiencies – delays in budget execution, mid-year budget revisions, and difficulties in mobilizing facilitators. The third extension (12 months) enabled the allocation of part of the project proceeds to support urban neighborhoods in West Sumatra recover from the September 2009 earthquake, but also accommodated some of the same inefficiencies that resulted in the second extension.
  • On balance, efficiency is rated substantial.

    a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


  • Rate Available?
    Point Value
    Coverage/Scope*
    Appraisal:
    No
    %
    %
    ICR estimate:
    No
    %
    %

    * Refers to percent of total project cost for which ERR/FRR was calculated

    6. Outcome:

    The project’s development objectives are highly relevant, and design relevance is substantial. Achievement of the three objectives – to establish or support representative and accountable community organizations that are able to provide services to the urban poor and increase the voice of the poor in public decision making; to make local government more responsive to the needs of the poor through increased cooperation with community organizations; and to improve services for the urban poor (financial services, social services and infrastructure) – were substantially achieved. On balance, efficiency is also assessed as substantial. Outcome is rated satisfactory.

    a. Outcome Rating: Satisfactory

    7. Rationale for Risk to Development Outcome Rating:

    There are two main risks to the project’s development outcome. First, central and local governments may withdraw their financial and technical support for the community development plans, which are the main instrument used by the community organizations established under the project to mobilize funds. However, both central and local level governments have indicated by their policies and actions a strong and long-term commitment to the financing of these plans. The Central Government has demonstrated its engagement to community-driven development programs, while local government partnership with, and financial support for, the community organizations benefiting from the project has exceeded expectations. This risk is considered low to negligible. Second, the ICR states (page 8) that “risks related to (a) adequate maintenance of the community infrastructure, (b) financial management practices, (c) procurement, and (d) availability of qualified facilitators were greater than anticipated, and not mitigated as successfully as expected.” Although the ICR reports that these shortcomings “did not fundamentally undermine achievement of the project outcomes,” the risk that they may do so in the future is rated as moderate.

    a. Risk to Development Outcome Rating: Moderate

    8. Assessment of Bank Performance:

    a. Quality at entry:
    Preparation had a number of strong points and weaknesses. The strong points were:

    • The ICR (page 13) reports that the Government considers the design of the project as a general model for community-driven development (CDD) programs in Indonesia, and has used many key features of the project for the preparation of its nationwide CDD program.
    • The project’s legal documents, by stipulating general principles and avoiding implementation details, allowed the latter to be based on a project manual prepared by the Government. This gave both the Borrower and the Bank the needed flexibility to modify implementation arrangements in response to changing circumstances.
    • In order to address governance issues and support effective anti-corruption policies, the Bank worked closely with the Ministry of Public Works (the implementing agency) in designing a complaints mechanism, counter-incentives, and sanctions when cases of fraud were proven.
    • The project was a follow-on to, and expansion of, the First Urban Poverty Project. This latter project had been modified by the Government during implementation with a view to transforming it into a mainstream poverty alleviation program. The team preparing the project under review adopted this mainstream approach, but built into the design three key lessons learned from the experience of the first project: (i) the need for additional support for the community based planning process, since it had become clear during the first project that more time was needed to build consensus with communities on the means to alleviate poverty. In this project, therefore, there was a much longer guided process for awareness-raising and socialization (4-6 months instead of less than two months); (ii) the need to establish clearer, standardized criteria, rules and principles for the operation of the revolving funds. The ability to support and administer such micro-credit facilities effectively at kelurahan level are severely limited. During implementation, as noted above, strict limits were placed on the proportion of grants allocated to micro-credit; (iii) the need to mainstream community development plans into local government policies and activities and to avoid the by-passing of established structures, which would hinder the building of productive and sustainable relationships between them and the communities. To this end, the project design included the poverty alleviation partnership grant component which was to lead to a strong degree of local government support and participation.
    The shortcomings were:
    • Although the proposed method of selecting participating kelurahans was relevant to the project’s poverty alleviation objectives (the criteria included not belonging to the 30% richest sub-districts in a participating district, and a poverty incidence of at least 20%), the project channeled “substantial” resources (the amounts are not specified) to neighborhoods that were not in fact urban. In principle, for a kelurahan to be included in the project, it had to be located in an eligible urban sub-district called a kecematan. The failure to follow this principle consistently led to the selection of kelurahans that -- although classified as urban -- were not located in a kecematan, and were, for all practical purposes, semi-urban or even rural neighborhoods.
    • Many of the outcome indicators in the PAD had neither baseline figures nor targets (see Section 10 below).
    • Not all risks to the project were identified or adequately mitigated: (i) despite the fact that many recent, Bank-supported projects in Indonesia had suffered delays due to avoidable delays due to late budget approvals and appointment of consultants, this did not feature as a risk in the PAD; (ii) to mitigate the risk that communities and local governments would be unable to work together, the PAD proposed cancelation of the poverty alleviation partnership grants should the risk materialize (which it did not), which would have deprived the project of the only instrument available for bringing the cooperation about; (iii) to ensure availability of proposed counterpart funding, the PAD proposed the principle of requiring counterpart funds to be used first; while sound in theory, this has proven very difficult to apply in practice; and (iv) in spite of ongoing problems with financial management and adequate management of infrastructure facilities in Indonesia, the risks of weaknesses in the monitoring of funds by kelurahans and inadequate maintenance of infrastructure were nonetheless rated as“negligible or low.” The Project Document for the Additional Financing contains a more thorough risk analysis with specific mitigation measures.
    • The establishment of stronger criteria and rules for the operation of revolving funds proved insufficient. During the first three years of implementation, there was limited compliance with financial management procedures and micro-credit loans continued to perform poorly. For the Additional Financing, therefore, it was decided to strictly limit the proportion of kelurahan grants allocated to revolving funds (see Section 4 above).

    Quality-at-Entry Rating: Moderately Satisfactory

    b. Quality of supervision:

  • The Bank mobilized a multi-disciplinary supervision team with expertise in project management, financial management, procurement, monitoring, and safeguards. There was good continuity, with one of the Task Team Leaders involved in preparation leading the project throughout the implementation phase.
  • Supervision was intense. The ICR reports (page 14) that in addition to the 13 formal missions over the eight-year implementation period, there were, on average, four site visits per year, involving 150 kelurahans. This effort enabled the Bank team to identify and proactively address key issues adversely affecting achievement of the project’s development objectives at an early stage, notably delays in the establishment of the MIS, budget execution and revision, consultant mobilization, facilitator recruitment, and limited compliance with financial management procedures of revolving funds.
  • The team was able, together with the Borrower, to achieve a fourfold expansion in project coverage and activities, supported by Additional Financing, in less than one year.
  • The team was also pro-active in changing the composition of the kelurahan grant allocations away from the relatively poorly performing micro-credits and in favor of social infrastructure, and in modifying funding rules to mobilize additional PAPG contributions from local government budgets.
  • The anti-corruption measures and the complaints handling system were regularly improved during implementation in close cooperation with the implementing agency.
  • The Bank was proactive in supervising the Borrower's MIS system, although the response was poor.

  • Quality of Supervision Rating: Satisfactory

    Overall Bank Performance Rating: Moderately Satisfactory

    9. Assessment of Borrower Performance:

    a. Government Performance:

  • The Government’s commitment to the project’s goals and to community-driven development in general has been demonstrated by its decisions to scale-up the project to nearly four times its original size, and by the measures it took to transform the project into a vehicle for delivering support to poor communities throughout Indonesia.
  • The ICR (page 14) notes, however, that the Government could have been more pro-active in preventing delays in budget execution and revisions which were among the principal causes of the extended implementation period.
  • The implementation of the project experienced a slowdown in the months following the national elections of 2004, mainly because of changes in personnel.

  • Government Performance Rating: Moderately Satisfactory

    b. Implementing Agency Performance:
    The implementing agency was the Ministry of Public Works. The Project Management Unit was housed in the Ministry.

  • The implementing agency was able, with the collaboration of the Bank team, to achieve a fourfold expansion in project coverage and activities, supported by Additional Financing, in less than one year.
  • The ICR (page 14) reports that the implementing agency supported a bottom-up, community-driven approach to poverty alleviation.
  • It was instrumental in establishing a comprehensive M&E system, which, although not problem-free, helped to maintain a high level of transparency of the outcomes of the project.
  • It took steps to modify implementation arrangements where needed and responded to complaints or suspicions about the misuse of funds with “great vigor.”
  • The ICR also notes, however, that there were several avoidable problems that might have been corrected by stronger implementing agency management, such as poor compliance by revolving funds with established financial management procedures, late payment and consequent high turnover of facilitators.
  • There were delays in the Ministry of Public Works’s recruitment of consultants to act as facilitators at the national and local levels. This was particularly marked at the outset, when, largely because of this, implementation started eight months late. Delays in consultant recruitment also meant that the Government’s survey of project results, based on key performance indicators, had still not been completed in time to be used in the ICR.

  • Implementing Agency Performance Rating: Moderately Satisfactory

    Overall Borrower Performance Rating: Moderately Satisfactory

    10. M&E Design, Implementation, & Utilization:

    a. M&E Design:

  • M&E design was centered on the development of a management information system that was “not only extremely comprehensive in terms of geographical coverage, but also required a large volume of data to be recovered” (ICR, page 8). The Ministry of Public Works was given responsibility for establishing and operating the MIS. The MIS was meant to cover not only the needs of the project under review, but also those of the First Urban Poverty Project and of the follow-up Third Urban Poverty Project. The intention was to have the MIS regularly updated, and for it to be publicly accessible through the Web. It would include a system for making, and dealing with, complaints. It was, however, over-ambitious in its complexity and data demands, while at the same time incomplete in its data coverage. The measurement of six out of eight outcome indicators and five out of 31 intermediate outcome indicators requires a survey. This was also the case for two output indicators included in the PAD. Many of the indicators in the PAD lacked baseline data and time-bound targets.

  • b. M&E Implementation:

  • The design of the MIS took considerably longer than expected. The system only became functional in mid-2005, and did not start to generate values of most key performance indicators until early 2006, over three years after effectiveness. As the ICR (page 9) notes, “the lack of reliable information compromised the system’s usefulness as a transparent M&E tool.” In an attempt to address these issues, the implementing agency assigned the responsibility for data accuracy and validity to the team of consultants tasked with data collection, and delayed payment until data was brought up to the required standard. “Together with spot check monitoring, these measures improved the quality of the information, but some problems with data collection continued to persist until the end of project implementation” (ICR, page 9).
  • Despite intensive supervision, the performance of the management information system remained less than fully satisfactory throughout the life of the project (ICR, page 14). Nonetheless, by closure, the MIS website was operational and provided access to information on the project, its predecessor and all follow-up operations. In 2009, the World Bank selected the website as the best from 42 submissions in the category “most innovative use of internet and cell phones” during the East Asia Pacific Innovation Days.
  • As noted in Section 4 above, an impact evaluation survey was conducted in three rounds between 2004 and 2007. Due to delays in appointing consultants, the results of a government survey based on an analysis of the project’s key performance indicators were still awaited when the ICR was completed.

  • a. M&E Utilization:

  • Despite continuing implementation problems, the ICR reports (page 9) that once the MIS had become fully operational, it was of assistance in (i) identifying and remedying issues of service quality; (ii) incorporating better controls as the project progressed; (iii) fine-tuning the targeting of resources to direct them where they were most needed and where they could have the greatest impact; (iv) ensuring transparency and strengthening public confidence in the openness and fairness of project resource allocation processes; and (v) demonstrating the tangible benefits the project brought to poor households.
  • The Government is using the MIS to monitor its nationwide community driven development program, which suggests that the system may be sustainable in the long run.

  • M&E Quality Rating: Substantial

    11. Other Issues:

    a. Safeguards:

  • The project was classified as Category “B” under OP 4.01 (Environmental Assessment). According to the PAD (page 30) and the Project Paper for the Additional Financing (page 2), four other safeguards policies were triggered: Involuntary Resettlement (OP 4.12), Indigenous Peoples (OP 4.20), Natural Habitats (OP 4.04), and Cultural Property (OPN 11.03).
  • The ICR provides very little information on safeguards, and there is no clear statement as to whether safeguards policies were complied with or not. The only discussion is the following (ICR, page 10): “Although some project activities triggered safeguards related to environmental assessment (OD 4.01), natural habitats (OP 4.04), cultural property (OPN 11.03), indigenous peoples (OD 4.20) and involuntary resettlement (OP 4.12), none of these activities caused significant adverse impacts. Given the small size of the sub-projects financed by the project (the cost of which often did not exceed US$2,000 equivalent), it was a priori expected that any adverse social and environmental impacts of such subprojects would be small or negligible. Indeed, the project generated substantial social and environmental benefits, especially through investments in drainage, water supply and sanitation. However, the recording of potential social and environmental impacts remained poor, and required significant improvement in the final stages of project implementation.” Subsequently the project team reported that "according to the project's latest aide memoire (Jan 2010), 'Overall implementation of the project shows that the sub-projects are in compliance with the environmental safeguards requirements.'" However, there is no such statement regarding the other other four triggered safeguards.

  • b. Fiduciary Compliance:

  • With regard to financial management, the ICR (page 10) states that “the project encountered problems with financial management throughout implementation, primarily because revolving funds were often unable to comply with FM guidelines, especially those related to safekeeping and the dissemination of financial reports. These problems were exacerbated by difficulties in finding qualified facilitators.” No further information is provided. There is no mention of external project audits or whether or not the auditor’s opinions were qualified. The project team provided the following information subsequently: "According to the project's latest aide memoire (Jan 2010), the government audit agency BPKP has completed its audits "on time and with clean opinions" ("clean" means "unqualified", in this context)." In June 2009, a financial management supervision mission stated: " In general, the mission found that, in most BKM [Community Organization or Badan Keswayadan Masyarakat] visited, financial records and supporting documents are in place. Although cases of misuse of fund and incomplete records were still found in certain BKMs and UPKs [Unit Pengelola Keuangan, the manager of funds for BKM]." Later missions reported similar findings. In other words, the project did confirm with the World Bank's fiduciary policies, but infractions with technical guidelines continued to persist in isolated cases."
  • There is no discussion in the ICR of procurement. The project team subsequently clarified that "There was no misprocurement of [Urban Poverty Project 2]-financed contracts. Most of the project's funds were channeled for community procurement, using the shopping method. These procedures were generally followed, although initially not all records of quotations and supply were properly documented, which required follow-up by Ministry of Public Works consultants and facilitators."

  • c. Unintended Impacts (positive or negative):
    The ICR (page 12) mentions two positive unintended impacts:

  • The project’s design was used to prepare two projects that provided emergency reconstruction assistance in rebuilding housing units and small-scale infrastructure destroyed by the 2004 tsunami and 2009 earthquakes in Aceh and North Sumatra.
  • Many local governments elected to co-finance implementation of community development plans from their own financial resources after the project’s kelurahan grants had been disbursed in full. Local governments continued to finance small-scale infrastructure from their own budgets, using planning and financing principles developed under the project.

  • d. Other:



    12. Ratings:

    ICR
    IEG Review
    Reason for Disagreement/Comments
    Outcome:
    Satisfactory
    Satisfactory
     
    Risk to Development Outcome:
    Negligible to Low
    Moderate
    Risks related to (a) adequate maintenance of the community infrastructure, (b) financial management practices, (c) procurement, and (d) availability of qualified facilitators were greater than anticipated, and not mitigated as successfully as expected. 
    Bank Performance:
    Satisfactory
    Moderately Satisfactory
    There were moderate shortcomings in quality at entry: not all risks were identified or adequately mitigated; the measures taken to strengthen financial management of the revolving funds proved insufficient; there were issues with kelurahan selection; and many of the outcome indicators had neither baseline figures nor quantitative targets.  
    Borrower Performance:
    Satisfactory
    Moderately Satisfactory
    Government could have been more pro-active in preventing delays in budget execution and revisions. Stronger implementing agency management could have avoided or mitigated problems such as poor compliance by revolving funds with established financial management procedures, late payment and consequent high turnover of facilitators, and delays in recruitment of consultants to act as facilitators at the national and local levels. 
    Quality of ICR:
     
    Unsatisfactory
     
    NOTES:
    - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
    - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

    13. Lessons:
    The ICR identifies a number of lessons from the preparation and implementation of this project. The key ones, with more general applicability to community-driven development (CDD) operations in Indonesia and elsewhere, are the following (with some adaptation):
  • Due to a high level of uncertainty at the preparation stage, flexibility in design and in implementation arrangements is of critical importance in large scale CDD projects. In this case, the Bank and Borrower were able to adapt to changing circumstances as new information became available.
  • While a comprehensive MIS helps to instill a culture of accountability and transparency at all levels of responsibility for project implementation, it can be over-demanding in terms of data requirements and of the need for accurate and timely reporting by communities and facilitators from the outset.
  • One of the main determinants of success in implementing CDD programs is the quality of the facilitators. In this case, many implementation difficulties might have been lessened or avoided if more had been invested in facilitator recruitment, training and remuneration.
  • The project demonstrated that financial incentives, coupled to intensive socialization and training of local government officials and their consultants, are an effective means to create a sustained interest of local governments in planning and co-financing small-scale infrastructure projects in partnership with communities.

  • 14. Assessment Recommended?

    No

    15. Comments on Quality of ICR:

    The quality of analysis in the ICR inevitably suffers from the fact that the government survey of project outcomes based on the key performance indicators was unexpectedly delayed, so that the results were unavailable as inputs to the document. This means that the ICR’s assessment of one of the development objectives -- to improve services for the urban poor (financial services, social services and infrastructure) – is less evidence-based than it should have otherwise been, and there are gaps in the evidence supporting the ratings for the other two objectives also. The brief presentation of safeguard and fiduciary dimensions gives no clear indication of whether there was compliance or not. The discussion of procurement is scattered throughout the report and not summarized or cross-referenced in the appropriate section. The date when the project extensions were granted are not given. The beneficiary survey is mentioned in the text but not summarized in the appropriate annex.

    a. Quality of ICR Rating: Unsatisfactory

    (ICRR-Rev6INV-Jun-2011)
    © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions