The Impact Evaluation: Objectives
1. Impact evaluations are an important tool for an institution to learn and make adjustments in its policies and projects and they are a true gauge of project sustainability. Impact evaluations serve several purposes, they: (i) complete the history of projects by writing the "concluding chapter"; (ii) assess the efficacy of the Bank and Borrowers by following investments through to their ultimate goals, and (iii) are a source of lessons needed to improve future development policies and appraisal methods.
2. The main objective of the evaluation study in this report was to understand the medium and long-term impacts (5 to 10 years after completion) of eighteen years of urban lending in Indonesia. The, projects included in the study [Urban I (Ln. 1040), Urban II (Ln. 1336), Urban III (Ln. 1653), and Urban IV (Ln. 1972)] were selected from the pool of completed Bank projects which have had sufficient time after the implementation phase for the outputs, effects and impacts to become evident.
3. This report presents the results of the evaluation of three components of the four urban projects: the Kampung Improvement Program (KIP) component in three major cities (Jakarta, Surabaya, and Denpasar); the sites and services program (Jakarta and Denpasar); and the citywide improvement program (Denpasar). In order to produce timely study results within a reasonable budget, our research focused upon a few impacts only. We studied four main areas of impacts: (i) improvements in the urban environment, (ii) changes in urban land values and markets, (iii) strengthening of institutions, and (iv) scope and durability of impacts (sustainability). 1/
Background to the Study (Chapter 1)
4. Indonesia has long been a recipient of Bank financing for its urban sector. Along with its growing prosperity, Indonesia's urban areas have also expanded considerably. The process of rapid urbanization started in the 1960s and early 1970s. The urban population increased from 22.6 million in 1970 to 32.8 million in 1980, to 55.4 million in 1990. Today, Indonesia's total population is estimated at 175 million persons, 28 percent of whom live in urban areas. With an annual urban growth rate of 4 percent, the country's urban population will comprise 36 percent of the entire population of the country by the year 2000.
5. Through the past three decades, the Bank has closely monitored this rapid urbanization. The first generation of Bank-financed projects included the four projects analyzed here -- Urban I, II, III, and IV (Table 1.1). The four projects were audited by the Operations Evaluations Department (World Bank 1983, 1986a, and 1994e, Box 1.2). The main component of these projects, and the main focus of our evaluation, was the Kampung Improvement Program. Kampungs, low-income dense urban areas, have been the target of several Bank-funded urban development programs (Box 1.1). The KIP component as a share of total lending in the four projects studied here ranged from 70 percent in Urban I to 32.8 percent in Urban IV. Other components in the four projects included improvements in solid waste management, drainage, community health services, land registration, and technical assistance and training. The goal of KIP was to alleviate poverty by supporting efforts to improve housing services and basic infrastructure in low-income areas known as kampungs. Although appraised in 1974 and subsequent years, that is, over 20 years ago, Urban I-IV addressed two crucial problems that still affect Indonesia's cities today: inadequate infrastructure coverage and deteriorating environmental conditions.
Low Cost Investment: Housing and Environmental Improvements (Chapter 2)
6. The KIP component of the Urban I-IV projects induced housing and environmental improvements for low-income urban households in Indonesia at a low cost of investment (ranging from US$118 per person in Jakarta to US$23 in smaller cities, 1993 US dollars). The study found that, given the generally favorable macroeconomic environment that prevailed throughout the project implementation period, and the positive demonstration effect of KIP, improvements in non-KIP kampungs have caught up with those in KIP kampungs. There is some evidence that improvements in non-KIP kampungs were completed at a slower pace when compared to the rapid and catalytic effect of KIP on those kampungs where it was implemented.
7. The most positive impact of KIP was the upgrade in the quality of life of kampung residents given by improved footpaths, lighting, education and health facilities, living space and reduction of housing density. In addition, we found that there is much wider access to clean and safe water (although a high number of people still use unsafe well water), private toilets/septic tanks (which were more accepted than the public washing and toilet facilities (MCKs), although not emptied regularly), and less frequent flooding outside their homes. More than two thirds of the respondents in project areas attributed the improvements in their kampung to KIP. The majority of respondents also suggested that today's overall environmental conditions in their neighborhood is better than before KIP was implemented but another third suggested that they were not completed satisfied. Garbage collection (both frequency and quality), for example, was cited as particular problem.
Persisting Environmental Degradation in Urban Areas (Chapter 3)
8. At the citywide level, environmental conditions have deteriorated particularly in areas where rapid population and economic growth has increased the demand for urban infrastructure services. Despite the improvements effected by KIP under Urban I-IV, the environment in major Indonesian cities continues to deteriorate, exacerbated by population growth. The urban environmental challenge in Indonesia today is far greater than it was when Urban I-IV were implemented, and much must still be done to head off the collision course between urban growth and environmental conditions: for example, Denpasar, an area of rapid urban growth, exhibited very bad solid waste management practices among the study areas. In other cases, drainage was not integrated with road upgrading causing flooding problems after roads were raised higher than house floors (Jakarta and Denpasar). In addition, drainage systems were not connected with broader infrastructure causing backlogs at the entrance of the city wide drainage systems, increasing flooding and breeding grounds for disease-carrying insects (Surabaya). At both the kampung and city-wide levels, the projects did not foresee some future environmental problems, such as increased traffic congestion and air pollution. And although the design standards of KIP, such as the width of footpaths, has helped increase access to fire-fighting units, the use of modern flammable building materials and overcrowding have increased the risk of fire.
KIP Impacts and Rapidly Emerging Real Estate Markets (Chapter 4)
9. Indonesian cities are changing rapidly and some kampungs are potentially valuable urban real estate, depending on their location within cities. The raising demand for prime land is driving kampungs into the modern market. The study attempted to address two project-related questions arising from the rapid transformation of urban areas: (i) are kampung residents benefiting from the investment? and (ii) does the demolition of improved kampungs signal the failure of the Urban I-IV projects? The study's findings were that residents did gain some benefits, but would have enjoyed more if their property rights had been more secure. The study also found that, because such large benefits flowed from KIP, initial project investments could still enjoy economic rates of return of 12 percent, even over a useful life of only five years. Some of the beneficial impacts of the Urban I-IV projects have been swept away by the demolition of improved kampungs to make way for modern urban development in strategic locations in many cities of Indonesia. The rising demand for urban land in a rapidly growing economy such as Indonesia's is likely to make the redevelopment of kampung land into commercial and up-market residential real estate increasingly common.
Community Voice in Structure Development: Urban Stability and Security of Tenure (Chapter 5)
10. The study's analysis of community participation/consultation in the planning and implementation phases of KIP showed that the involvement of residents varied from one kampung to another. In some kampungs, residents were consulted about the convenience of the location of roads and footpaths, and slightly less about the convenience of the location of water standpipes and sanitary facilities. Consultation in the planning phase of KIP took the form of local meetings in the local officials offices (kelurahan). Similarly, the degree of participation varied. In some kampungs, residents worked through local neighborhood associations to help with simple aspects of KIP implementation, such as the demolition of fences or dwellings. In other kampungs, residents merely observed construction activities.
11. KIP did not encourage an influx of higher-income groups (that is, "gentrification") into the kampungs, as had originally been feared. In fact, KIP did not disturb the existing, residential stability of the kampungs although the social profile of the kampungs has changed under KIP. Residents are better educated and healthier; household sizes have declined; more residents are employed and have greater income; and women have taken a more active role in meeting the economic needs of their families. Improvements in population conditions are not only caused by KIP but also by the opportunities generated by economic growth.
12. Although KIP did not attempt to influence land and housing tenure directly, the expectation was that it would increase ownership; as community security increased, more people would be motivated to clarify and improve the status of the land they occupied. The study found that KIP did in fact increase ownership throughout the improved kampungs or at least greater security of tenure. In addition, the stronger sense of tenure gave residents an incentive to participate actively in the operations and maintenance, O&M, of community facilities, although O&M levels vary from place to place and it is considered one of the weakest points of KIP.
Institutions: What Did They Learn and Remember? (Chapter 6)
13. Despite their focus on physical improvements in the infrastructure and the environment, Urban I-IV did have some landmark impacts on institutional development in the urban sector of Indonesia. For example, new agencies were created by the Government of Indonesia to direct development and investment activities, and the managerial capabilities of existing agencies were strengthened considerably. In particular, the National Urban Development Corporation (PERUMNAS) was established as a new government agency for managing low-cost housing development, and the state savings bank (Bank Tabungan Negara, BTN) was allowed to introduce mortgage lending operations for the first time in Indonesia. And these impacts have been sustained, insofar as the agencies remain important players on the urban stage in Indonesia to this day.
14. Thus, one of the lasting impacts of all four projects was to keep important institutional development issues -- notably cost recovery and decentralization -- on the urban sector reform agenda. On the issue of cost recovery, government agencies have adhered consistently to the principles of cost recovery embodied in successive urban projects but there are strong underlying reservations about how and when effective cost recovery should be achieved. Despite the recent demonstration of the limits reached by a centralized model in meeting urban sector needs, the government remains the dominant decision-maker in urban development to this day. Yet difficulties experienced by both the government and the Bank in supervising multi-city projects from Jakarta hastened efforts to devolve project implementation responsibilities to the local level of administration.
15. The study highlighted, however, that the rich learning experience of the urban projects was neither well-documented nor followed progressively throughout the implementation period. Although Monitoring and Evaluation, M&E, was emphasized after the first two projects, there was a failure to implement the agreed M&E component.
Findings and Recommendations (Chapter 7)
16. The main finding of this report is that the Kampung Improvement Program improved the quality of life of Indonesian urban areas at a low cost of investment. The projects had immediate and very positive impacts on the kampungs where the inputs were targeted.
17. Based on both primary and secondary data, the evaluation yielded the following specific findings:
- Improvements in the kampungs prompted residents to invest more in home repairs and in the operations and maintenance of community infrastructure.
- The immediate areas surrounding the KIP kampungs and the site and services program areas benefited from the projects, as the physical and economic impacts of KIP and sites and services programs rippled to those neighboring communities.
- On the other hand, at the citywide level, environmental conditions have deteriorated, particularly in areas where rapid population and economic growth has increased the demand for urban infrastructure services (housing, water and sanitation, footpaths, roads, solid waste disposal, and drainage). Thus, KIP kampungs and site and services program areas and their immediate surroundings are islands of environmental improvements
- Community consultation and participation in the early stages of project preparation and design was important for instilling a sense of project ownership by the community.
- KIP did not encourage urban mobility while creating a greater sense of tenure security towards house ownership. KIP's rapid and extensive coverage may partially account for the low level of household turnover or mobility. City residents, observing that KIP was extended to most of the city, did not feel it necessary to move to take advantage of the program.
- The rising demand for scarce urban land in major Indonesian cities is likely to make the redevelopment of kampung land into commercial and up-market residential real estate increasingly common.
- Despite their focus on physical improvements in the infrastructure, Urban I-IV did have some landmark impacts on institutional development in the urban sector of Indonesia, through the creation of new agencies and the strengthening of existing institutions at the central level.
18. According to the previous findings, the following recommendations and lessons can be proposed:
- An upgrade program should integrate flood control measures and drainage networks at the citywide scale. Improved drainage systems helped to reduce flooding. This is a direct positive impact of KIP investments. However, some complaints were voiced about drainage backlogs at the entrance to the main city drain and backups when drains are not cleaned regularly. Improvements in one area should not have a negative impact on residents in neighboring areas.
- Some aspects of the infrastructure and environmental conditions can be controlled and managed by communities -- for example, dwelling conditions, road and footpath maintenance, solid waste collection from houses, and the upkeep of local drains. Other aspects, such as the illegal dumping of solid waste, the integration of kampung infrastructure with the city-wide infrastructure and the effects of pollution, are beyond the control of the communities and need to be controlled and managed by local governments with assistance by central national authorities.
- The design of urban projects in Indonesia should include an assessment of the dynamics of the local real estate market. Growing demand for modern real estate development in Indonesia's cities can thwart low income infrastructure improvements made by projects in prime locations. Redevelopment of many kampungs may come sooner than later, in which case, future projects need to address up-front policy issues of compensation for displaced low income families.
- Future projects should promote working partnerships with community groups and nongovernmental organizations (NGOs) as a way to ensure that they are responsive to O&M requirements in their kampungs. Consultation and participation of communities and NGOs in the early stages of project preparation and design should be included at the planning stage, since these aspects are important for instilling a sense of project ownership by the community. Although the concept of community consultation and participation is now widely accepted, it still means different things to different people, and many of the local government staff understand the concept but do not understand how it translates into practice. Therefore, greater understanding and better guidelines for community members, NGOs and project staff should be established
- It is important that the institutional memory and lesson learning capacity developed for urban development projects be maintained so as to provide guidance for future endeavors and to facilitate the sharing of knowledge of these endeavors world-wide. Important steps must be taken to consolidate institutional memory within both the government and the Bank in future projects. With the support of the Bank, BAPPENAS plans to develop an in-house evaluation capability in all sectors, including impact evaluations. Particularly important is ensuring that baseline data and evaluation systems are in place prior to initiation of any project.
1/ As KIP is an urban infrastructure program that did not deliver completed shelter units, our study does not cover all aspects of housing in Indonesia. Important issues, such housing markets and the roles of the public and private sectors are discussed in OED's recently completed audit of the Indonesia: Housing Sector Loan (Loan 2725), (World Bank, 1995).