Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Cl Santiago Urban Transport Technical Assistance Project


  
1. Project Data:   
ICR Review Date Posted:
11/08/2013   
Country:
Chile
PROJ ID:
P086689
Appraisal
Actual
Project Name:
Cl Santiago Urban Transport Technical Assistance Project
Project Costs(US $M)
 6.00  4.38
L/C Number:
L7316
Loan/Credit (US $M)
 4.80  2.37
Sector Board:
Transport
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  07/05/2005
 
 
Closing Date
12/31/2009 12/31/2011
Sector(s):
Central government administration (72%), General transportation sector (25%), Sub-national government administration (3%)
Theme(s):
Pollution management and environmental health (25% - P) Other urban development (25% - P) Urban services and housing for the poor (24% - P) Social analysis and monitoring (13% - S) Improving labor markets (13% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Victoria Alexeeva
Kristin Hallberg Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The statements of objectives in the Project Appraisal Document (PAD) and the Loan Agreement are not identical. The project development objectives in the PAD are " to support the implementation, monitoring, evaluation and continuous planning of the urban transport reform program and to strengthen the capacity and improve the procedures to mitigate potential adverse impacts of transport infrastructure works". The statement of objectives in the Loan Agreement is "to support the Borrower in achieving an efficient and sustainable urban transport system for Metropolitan Santiago". The achievement of the objectives will be evaluated based on the statement in the Loan Agreement as it is a legally binding document.

The project underwent a restructuring in October 2009. The main changes were (a) modification of the project scope; (b) change in implementation arrangements; (c) replacement and fine-tuning of indicators; (d) extension of closing date; and (e) alignment of PDO in the PAD and Loan Agreement.

The original PDO indicators were changed from (i) completing a survey to assess Transantiago’s impact on transport demand; and (ii) designing a monitoring system for Transantiago’s environmental impact to (i) improvement measure identified in the evaluation of Transantiago implemented and (ii) information on Transantiago’s impact on air quality available.

A number of outcome targets were revised as follows:

  • "Mitigation of negative effects of Transantiago on low-income groups" was changed to " Mitigation of negative effects of Transantiago on street vendors".
  • Dropped: "Improved road infrastructure management", "Transantiago users well informed about the system", and "Better coordination between transport and land use planning".
  • Added: "Improvement in quality and/or efficiency of Transantiago"; and "Improvement of Transantiagos financial/fare control and management system".

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
Component 1: General Urban Transport (appraisal cost US$1.07 million; actual cost US$0.8 million). The component aimed to (i) obtain qualitative and quantitative data on urban passenger movements; (ii) forecast urban transport demand by incorporating socio-economic variables into the transport planning model ESTRAUS; (iii) lay the basis for Transantiago’s long-term institutional design; and (iv) implement a road infrastructure maintenance management system.

Component 2: Public Transport (appraisal cost US$1.81 million; actual cost US$2.34 million). This component aimed to (i) obtain information to optimize fares; (ii) design intermodal stations; (iii) design a graphic information system; (iv) obtain information to optimize travel times; and (v) inform users of the new system.

Component 3: Environment (appraisal cost US$0.23 million; actual cost US$0.43 million). The component aimed to provide regular data on air quality and noise levels necessary to monitor and maximize the positive impacts expected from Transantiago.

Component 4: Land Use (appraisal cost US$0.89 million; actual cost US$0.57 million). The component aimed to (i) update the sub-models of a mathematical model MUSSA and assess three road pricing alternatives; (ii) promote activity concentration around transfer stations; (iii) facilitate transport and land use planning coordination; and (iv) support the implementation of the urban transport impact evaluation system (SEISTU) to ensure that real estate developers pay for their negative transport externalities.

Component 5: Social Aspects of Transport (appraisal cost US$1 million; actual cost US$0.21 million). The component aimed to (i) support a training program for transit workers and (ii) analyze Transantiagos impact on low-income groups.

Component 6: Mitigation of Potential Adverse Impacts of Transport Infrastructure Works (appraisal cost US$0.28 million; actual cost US$0 million). The component aimed to (i) strengthen Chiles capacity in the areas of involuntary resettlements and (ii) harmonize environmental guidelines.

Component 7: Project Management (appraisal cost US$0.5 million; actual cost US$0.02 million). The component supported strengthening of the project coordination unit (PCU) and carrying out of project audits.

As noted above, the project underwent a restructuring in October, 2009. The restructuring eliminated the original Component 6 and changed several project activities within the existing project components. It also ensured consistence in substance and language between the components and project activities in the project documents and the Loan Agreement. The revised components were the following:

Component 1: General Urban Transport: to (i) more precisely forecast urban transport demand by incorporating socio-economic variables into ESTRAUS and (ii) lay the basis for Transantiagos institutional design over the long term.
Component 2: Public Transport: to (i) carry out evaluations of Transantiagos performance and identify measures to improve the public transport system; (ii) improve Transantiagos financial and fare management system; and (iii) improve the quality and /or efficiency of public transport in Metropolitan Santiago.
Component 3: Environment: to monitor the impact of Transantiago on air quality.
Component 4: Land Use: to (i) update the sub-models of MUSSA and assess three road pricing alternatives and (ii) support the implementation of SEISTU to ensure that real estate developers pay for their negative transport externalities.
Component 5: Social Aspects of Transport: to analyze Transantiagos impact on street vendors.
Component 6: Project Management: to strengthen the PCU and to carry out project audits.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost: Total project cost was US$4.38 million at closure compared to the appraisal estimate of US$6 million due to project restructuring and a partial loan cancellation in the amount of US$1.98 million as there were no longer needed and to grant a four-month grace period.

Financing: The estimated World Bank Group contribution was US$4.80 million, which reduced to US$2.37 million at closure due to project restructuring. The loan was partially cancelled in the amount of US$1.98 million.

Borrower contribution: The actual Borrower contribution was US$2.01 million, as compared to the estimated amount of US$1.2 million, as local resources became available during project implementation to finance
larger portions of the project activities than had been envisaged.

Dates: The project was extended by two years from the original closing date of December 31, 2009 to December 31, 2011 to complete the activities revised during the restructuring of October 2, 2009.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High.
At the time of appraisal, Santiagos transport system suffered from serious problems that included exponential increase in private car use, steady decrease in public transport modal share, deficiencies in the bus system organization and high bus related accident levels, and high air pollution levels with transport as the major contributor to local emissions. The project development objective was consistent with the Government of Chile's public transport reform priorities outlined in the 2000-2010 Santiago Urban Transport Plan (PTUS). The
Presidents Government Program (2010-2014) advocated for "a dignified and efficient public transport". The Transantiago pubic transport reform was among the strategic objectives of the Transport Minister who promised substantial improvements in service quality to the 2.5 million bus users in Santiago. The objective remained relevant to the World Bank Group's FY11-16 Country Strategy for the Republic of Chile at project closure, which focused on public sector modernization, job creation and equity improvement, and promoting sustainable investments. The objective was also in line with the main priorities of the Country Strategies at appraisal and during implementation, which aimed at sustaining overall economic growth and social progress, heightening inclusion, especially of rural populations and vulnerable groups, and modernizing the state as the underpinning for development.

b. Relevance of Design:
Substantial.
The statement of development objective was clear "to support the Borrower in achieving an efficient and sustainable urban transport system for Metropolitan Santiago". The link between the activities financed by the project and the outputs and outcomes related to the attainment of the development objectives was logical. For example, enhanced efficiency of Santiago's urban transport system was supported by the activities supporting improvement of the transport planning model and Transantiago's financial and fare management system, assessment of the road pricing alternatives, and laying the basis for Transantiago's institutional design over the long term. Enhanced sustainability of Santiago's urban transport system was supported by the activities aimed at monitoring the impact of Transantiago on air quality and support for the impact analysis on street vendors.


4. Achievement of Objectives (Efficacy) :

A. Enhanced efficiency of urban transport system for Metropolitan Santiago: Substantial.

Outputs
  • A socio-economic sub-model for transport planning model ESTRAUS was developed for the Transport Planning Secretariat SECTRA as planned.
  • A study to analyze possible institutional structures for Transantiago, propose a new institutional framework, and devise an action plan to implement it was carried out by the Borrower. The project complemented this activity by supporting the organization of an international conference on Metropolitan Transport Authorities.
  • The entity in charge of the urban transport reform and project coordination unit "Transantiago-SE" commissioned a study to restructure the Ministry of Transport and Telecommunications, to which it belongs as a temporary program. The recommendations laid the basis for the transformation of "Transantiago-SE" into an autonomous agency with legal personality.
  • The project supported an organizational study that reviewed the internal organization, processes, systems and resources of Transantiago-SE, defined a strategy, functions, staffing and other needs, and redesigned the business processes.
  • The study to collect, through household panels, qualitative and quantitative data on mobility patterns and analyze the changes in these patterns in the light of different urban transport projects and policies, was cancelled. The Borrower was contracting an O-D survey with local resources.
  • The assessment to improve the public transport system in Santiago was carried out. The main recommendations from this assessment pointed to (i) fare increases, (ii) operational/service improvements, mainly to reduce the excess supply, short routes and transfers, (iii) changes in the contracts with the bus operators to create incentives to provide good quality services, and (iv) fare evasion reduction. The assessment also made it clear that Transantiago-SE lacked the tools to adequately monitor the service offer and demand. As a follow up, "Transantiago-SE", with their own funds, contracted the consulting firm to develop a transport model with data on load profiles, origins and destinations and generate other data to plan new operational improvements.
  • A study to evaluate the bus depot and terminal infrastructure, its management, and prepare an integral management plan was concluded after project closure. At the time of ICR preparation, "Transantiago-SE" set up a unit to deal with this topic and implement the study recommendations.
  • A study to develop a methodology to assess the economic and financial health of the bus operators was completed. "Transantiago-SE" also hired a consultant to make urgent short term improvements to the fare and financial management system, which made the system more reliable and stable. At the time of ICR preparation, "Transantiago-SE" was preparing the bidding documents for a second study mainly to adapt the fare and financial management system to the new payment mechanism in the renegotiated contracts with bus operators.
  • The project financed the identification of international good practice cases in public transport as well as two study trips of urban transport professionals from Santiago to Colombia and Spain. These visits revealed good practices, which culminated in the design of several pilot projects.
  • Two pilot projects were completed and evaluated at the time of ICR preparation: (i) for testing different colored pavement surface treatments for permanent bus lanes, and (ii) for photovoltaic lighting installation at 263 bus stops. The third pilot project to test traffic signs with variable speed information in school areas could not be carried out because Transantiago-SE received only one offer in the procurement process for equipment and the respective process had to be cancelled. Transantiago-SE will carry out the pilot project with its own resources. The pilot project to test efficient driving was postponed because the application of the colored pavement treatment and the control cameras in bus corridors to be used for this test will only be ready in the second half of 2012.
  • The study to review the fare system after the introduction of Transantiago was eliminated from the project scope because the the origin-destination (O-D) data and information on travel patterns essential to carry out the study as originally planned was not expected to become available in time to complete the study before loan closure. "Transantiago-SE" analyzed and revised the fare system on its own.
  • Transantiago-SE" implemented a number of activities with local resources and thus they were eliminated from the project scope during project restructuring, i.e., design of intermodal stations, pilot testing of Transantiago's information system, and information campaigns.

Intermediate Outcomes
  • The socio-economic sub-model of ESTRAUS is crucial in transport planning to more precisely forecast travel demand since the socioeconomic characteristics of passengers determine their travel behavior. SECTRA used the tool to plan the extension of the Santiago metro system and to follow up on the 2006-2012 development plans for the transport system in Metropolitan Santiago. At the time of ICR preparation, SECTRA was planning to update the forecast tool with the results of the origin destination (O-D) survey ongoing.
  • Based on the institutional analysis and the results of the conference, in 2007, the Borrower submitted a draft law on the creation of a Metropolitan Transport Authority for Santiago to Congress. At the time of ICR preparation, the draft law was still pending.
  • At the time of ICR preparation, the Government of Chile was carrying out the preparatory activities to implement the transformation of "Transantiago-SE" into an autonomous and independent agency.
  • The results of the organizational study were the basis for numerous improvements in terms of processes, procedures, management control, project management, system architecture, and human resources. For instance, as a consequence of the study, the process to generate performance indicators was fully automated. The revision of the internal processes also helped to eliminate duplications, such as the generation of operational indicators, which were collected by the operations unit and an external consultant. This strengthened Transantiago-SE as an entity in charge of the urban transport system in Santiago.
  • The O-D survey ongoing at the time of ICR preparation is expected to provide data and information on the impact of Transantiagoon transport demand.
  • Transantiago-SE made some initial improvements to the fare and financial management system. In the future, it is likely that the data regarding the system’s revenues and expenses will become available in real time. According to the renegotiated contracts, however, data on some of the penalties or the quality bonus is only assessed at certain intervals. Therefore, it will not be possible to make the information on the systems deficits/ surpluses and payment obligations available in real time/on a daily basis.
  • Both completed pilot projects showed potential for public transport quality and efficiency improvements. As a result of the 1st pilot project, at the time of ICR preparation, Transantiago-SE was preparing the bidding documents to apply the colored pavement surface treatment on 50 km of permanent bus lanes. It was also testing the effectiveness of applying intermittent signs on the pavement to better signify bus lanes which only function during certain hours of the day. As a result of the 2nd pilot project, at the time of ICR preparation, Transantiago-SE signed a contract for the installation of photovoltaic lighting at 1,000 additional bus stops. Transantiago-SE also planned to pilot test the use of the excess energy from the system to operate a real time information system on bus arrivals.

Outcomes
  • The recommendations of the public transport system assessment were implemented as follows:

(i) Between 2010 and the time of ICR preparation, bus fares increased from CLP400 to CLP580, Metro fares during off peak hours from CLP400 to CLP600, and Metro fares during peak hours from CLP460 to CLP660. These fare increases reduced the operating subsidies from 49% and 47% of the system’s total costs in January 2009 and 2010, and to 35% and 37% in January 2011 and 2012, respectively;

(ii) The direct bus service 118 was created from Maipú to La Florida in 2011 that provided nearly 3 million annual passengers with a direct origin-destination connection. The new direct bus service 125 from Lo Espejo y PAC to the center of Santiago decreased transfers on this route by 38% annually. Additionally, the total length of bus routes increased from 2,672 km in 2010 to 2,737 km in 2012 (by 2.4%), thus enhancing network coverage. More operational improvements were going to be implemented in the framework of the renegotiated contracts with the bus operators.

(iii) The contracts with bus operators were renegotiated and came into force between March and June 2012. These contracts introduced incentives for improved service quality (e.g. new remuneration criteria for bus operators, which are 70% based on passengers transported and 30% based on km provided, remuneration linked to service quality indicators and a bonus for good service quality), gave operators a stronger role in the fight against fare evasion, and provided the necessary flexibility to make additional operational improvements. The information on the profitability of the contract modifications with the bus operators was also fundamental in the contract renegotiations since it enabled a reduction in the contract price.


B. Enhanced sustainability of urban transport system for Metropolitan Santiago: Substantial.
Outputs
  • The first study to assess Transantiago’s impact on the air quality in Santiago was prepared, including the design of a methodology. In 2009, "Transantiago-SE" financed the follow up study with their own resources.
  • The planned activities on updating the Land Use Model for Santiago (MUSSA) were carried out, including a social assessment of three congestion pricing alternatives in the framework of Transantiago.
  • The study to improve the legal framework and information system of the Urban Transport System Impact Evaluation System (SEISTU) was prepared.
  • A road infrastructure conservation and management study that was expected to result in the implementation of a planning system for urban road maintenance was cancelled.
  • The assessment of the impact of Transantiago on street market vendors was carried out.

Intermediate Outcomes
  • By developing a tool for strategic transport and land use planning as well as strengthening SEISTU, the project laid the basis for better urban transport planning. The blueprint to strengthen SEISTU was prepared. A working group was set up with representatives of the Ministry of Housing, the Ministry of Transport and Telecommunications and the Transport Planning Secretariat SECTRA to implement a new legal framework for SEISTU. The activities of the working group were suspended because the revision of the legal framework was no longer a government priority at the time of ICR preparation.
  • The results of the Transantiago impact study on street vendors were used to modify bus routes and bus services during the operating hours of street markets. At the time of ICR preparation, the Ministry of Economy was drafting a law that used the results of the project financed study of street market vendors as input. It is likely that this new law will lead to the implementation of measures to improve the general condition of street market vendors, which were proposed in the study but are outside the responsibility of "Transantiago-SE".
  • With the preparation of the Infrastructure Master Plan for Public Transport 20112015, which places strong emphasis on road maintenance and traffic management measures, "Transantiago-SE" created a simple information system in Excel spreadsheet format with the pavement status and maintenance needs. The system is maintained with information received from bus operators, the municipalities and collected by "Transantiago-SE" through field visits. The system is the basis for the decisions on pavement maintenance carried out by the Regional Service for Road and Urban Development (SERVIU).

Outcomes
  • The air quality impact monitoring study showed that Transantiago was responsible for important decreases in particulate matter (PM), Nitrogen Oxide (NOx), and soot at street level. The results of this and a follow-up study were used to make the environmental requirements of public transport vehicles more stringent, which led to additional NOx and PM reductions in 2011 as compared to 2010, thus contributing to the system’s environmental sustainability. The requirements were set out in the Environmental Prevention and Clean-Up Plan (Plan de Prevención y Descontaminación Ambiental) and in the MTT emission norms revised in 2009. In 2011, Transantiago emitted 77.2 tons/year of PM and 4,211.7 tons/year of NOx, which was 19.9% less in PM and 1.7% less in NOx compared to 2010.

I
The ICR acknowledges that the achievements of Transantiago were largely attributable to the Government of Chile, while the project contributed within its limits of scope. Besides, a number of studies complementing the project's activities were carried out by the Borrower on its own through local contracts and finances.

5. Efficiency:

No economic and financial analysis was carried out at appraisal.


The prices of studies carried out were generally below the respective cost estimates. The contracted studies were quickly implemented and there were no cost overruns and only few contract modifications. In the case of Components 1 and 3, the cost increased due to the inclusion of additional activities not originally budgeted. The overall project cost was lower than the appraisal and restructuring cost estimates.

The project was completed with a two-year delay as a result of the restructuring due to exogenous events. Efficiency is assessed as substantial.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The project’s development objective was highly relevant to the challenges faced by the public transport sector in Metropolitan Santiago. Design is rated substantial. The project substantially contributed to better efficiency and sustainability of the urban transport system: the Government of Chile acted with a number of concrete reform measures that reduced transfers, enhanced network coverage, and increased operating cost coverage. The information on Transantiago's air quality impacts was used to revise and make environmental standards of public vehicles more stringent. Efficiency is rated substantial despite a two-year delay that was due to exogenous factors. Outcome is assessed as Satisfactory.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:

There is a strong political support of the urban transport reform as spelt out in the current Presidential and Ministerial Programs.

The improvements to the quality and efficiency of public transport deriving from the large scale implementation of the pilot projects will be most likely sustained. It is unlikely that the improvements made to Transantiago’s financial and payment system will be reversed, in particular considering that by the ICR preparation the agency in charge of urban transport reform implementation "Transantiago-SE" was preparing the bidding documents to further improve the system. Also, as the ICR states, there was a political will to provide 'Transantiago-SE' with more autonomy.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:
The Banks support to the implementation of the 2000 - 2010 Santiago Urban Transport Plan was expected to help address the main issues affecting transport and mobility in Santiago.This support came in a package of three instruments: this TAL (P086689), the Santiago DPL, and the Sustainable Transport and Air Quality Project for Santiago financed through a Global Environment Facility grant (GEF project P073985). The technical design of the technical assistance loan benefited from the Banks involvement in the GEF project and in the parallel Santiago DPL.


The preparation lasted approximately two years and was done together with the preparation of the Santiago DPL. The Bank team involved in project preparation had a broad skill mix, including various seasoned and experienced transport and urban transport specialists, environmental and social specialists as well as safeguard and fiduciary specialists. The technical preparation was thorough. The TORs for more than half of the activities were ready at the time of appraisal. The Bank team also adequately handled the fiduciary and safeguard aspects.

Despite the thorough technical preparation, there were shortcomings in risk identification and mitigation. The comprehensive risk assessment for this project did not anticipate the launch difficulties of Transantiago and the shift in project priorities due to these difficulties and the change in governments. Since these risks were not identified, no mitigation measures were designed and applied. The project activities were described in a precise and detailed manner leaving no flexibility for changes. The shift in such priorities necessitated restructuring of the project. There were also some shortcomings in the results framework (see section 10 below).

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:
The team had a high level of involvement throughout the lifetime of project implementation. Supervision missions took place twice a year. The Bank team maintained an adequate skill mix, including transport, environmental and fiduciary specialists. The team was in constant contact with the PCU to provide international experience, good practice examples and technical advice, review and comment on terms of reference and project reports, and help address implementation issues. Because of the rigid description of the project activities and the great need for flexibility in this operation, the team showed creativity to solve implementation problems and adapt the project activities to changed local conditions.

After the introduction of Transantiago, which almost completely stopped project implementation during one year, Bank supervision was even more intensified through monthly virtual supervisions via conference calls documented in detailed minutes. This together with project restructuring brought project implementation under control again.

During the last few years of project implementation, Bank supervision put a lot of effort in defining the pilot projects and pushing for their implementation before project closure. Safeguard and fiduciary aspects throughout the project implementation period were adequately taken into account.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The Government of Chile (GoCH) involved a broad range of sector agencies in project preparation, assigned the necessary staff and resources, and put an enabling environment into place. During project implementation, the GoCH provided more counterpart funds than agreed upon and stuck to the institutional arrangements defined at appraisal.

The GoCH showed a high level of commitment in the creation of an efficient and high quality public transport system and consequently to the achievement of the PDO. The commitment to Transantiago grew even stronger after the launch of the new system, which had caused enormous distress to users and risked government collapse. At that time, the GoCH did everything possible to correct the launch mistakes and make Transantiago work. The dedication to fix Transantiago’s problems affected the resource priorities within Transantiago-SE, led to several changes of Transantiago-SE’s coordinator and key staff, and heavily slowed down project implementation for at least a year. Once the initial difficulties of Transantiago were overcome, the GoCH was determined to use the restructured TAL as a tool to improve the public transport system and build capacity and skills within Transantiago-SE.

As ICR notes, at the time of the preparation of this ICR, Transantiago has become one of the best quality public transport systems in Latin America. Most notably, it managed to: (i) provide high accessibility at a reasonable fare; (ii) fully integrate the bus and Metro services in the city; (iii) introduce a contactless smart card with 100% penetration; (iv) formalize and professionalize bus operations; and (v) drastically reduce the significant externalities that Santiago used to face in terms of pollution, accidents and noise. These achievements were huge and largely directly attributable to the Government of Chile.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:
The project was implemented by a dedicated Project Coordination Unit under the aegis of "Transantiago-SE", the entity in charge of the urban transport reform. The PCU had a small, well-qualified and highly committed team in charge of both the TAL and the GEF projects. In technical matters, the unit was supported by technical staff from other areas of Transantiago-SE, Transport Planning Secretariat- SECTRA, and the Ministry of Transport and Telecommunications. They did an exemplary job in terms of preparing TORs and technical specifications and supervising the studies and project activities. Study reports were thoroughly reviewed normally by at least three people. The team provided excellent comments and high-quality technical contributions. For most studies, they requested further analyses, which implied additional supervision work. The team also provided valuable technical input to project restructuring and was highly committed to the successive design of the pilot projects.

In safeguard and fiduciary terms, the performance of the PCU was also satisfactory, except for minor accounting errors that did not imply any accountability issues, and some shortcomings in procurement mainly due to the temporary unfamiliarity of the team with Bank procurement rules and procedures after the sudden departure of the head of the PCU in 2009.

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
Both original and revised key indicators included baseline data and precise targets set at appraisal (PAD Annex 3; ICR-Annex 2 and Section F of the Datasheet). For example, the original indicators included (i) completing a survey to assess Transantiago’s impact on transport demand, and (ii) designing a monitoring system for Transantiago’s environmental impact. The revised indicators were to (i) improvement measure identified in the evaluation of Transantiago implemented and (ii) information on Transantiago’s impact on air quality available.

The key indicators measuring the achievement of the project objectives, however, reflected more outputs rather than outcomes. Mere production of outputs related to institutional strengthening is not a sufficient evidence of tangible improvement in the functioning of the urban transport system.

"Transantiago-SE" was responsible for managing and operating the M&E system.

b. M&E Implementation:
As part of the project restructuring, several indicators and outcomes were revised to align them with the project modifications. Also, the PDO wording of the Loan Agreement was adopted since it was considered a better reflection of the intended objective. One of the indicators that measured the improvements in the financial and fare management system became redundant.

a. M&E Utilization:
The M&E system was used to track project progress and inform management decisions on restructuring of the project. The restructuring realigned the project activities with the changed needs and priorities of the Borrower and brought the project back on track to achieve its PDO.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
At appraisal, this was a Category "C" project and no safeguards policies were triggered. In the framework of project restructuring in 2009, the Borrower decided to change the project scope to include the execution of pilot projects, which were to be defined in the course of project implementation. Since these pilot projects could have required small works or have had other negative environmental or social implications, the projects environmental category was changed to B and OP4.01 Environmental Assessment was triggered.


Environmental Assessment and Management: "Transantiago-SE" prepared, and the Bank approved, simplified environmental guidelines. These guidelines included the description of the type of works envisaged in the pilot projects and their potential environmental impacts, the applicable environmental framework and norms, mitigation measures, institutional responsibilities, and a checklist for screening, among others. "Transantiago-SE" screened all pilot projects for potential negative environmental and social impacts. For the pilot project to test colored road surface treatments for bus lanes, "Transantiago-SE" provided the Bank with an assurance from the provider of the road treatments that they did not have any negative environmental impact. In the case of the pilot project to install solar lighting at bus stops, "Transantiago-SE" produced satisfactory evidence on environmentally sustainable recycling procedures for batteries. No negative social issues were anticipated or arose in these pilot projects.

b. Fiduciary Compliance:
The ICR states that "fiduciary aspects overall were handled in a satisfactory manner even though the project registered minor shortcomings in FM and procurement".

Procurement: In the Independent Procurement Review carried out in March 2009, procurement was considered moderately satisfactory, mainly because of temporary weaknesses in internal controls and the implementing agencys low capacity to conduct procurement as per Bank guidelines. This was due to a sudden departure of the then PCU head. The procurement post review carried out at the end of 2010 considered procurement as moderately satisfactory because of the limited amount of procurement activities carried out during the review period. The last procurement post review at the end of 2011 considered procurement fully satisfactory.

Financial management. The project maintained overall satisfactory performance and low risk. Timely and reliable interim un-audited reports have allowed managing and monitoring project implementation. Audit reports were by and large submitted on time and considered acceptable to the Bank. The 2006 to 2008 audit reports had qualified opinions because the accrual basis was used to register a payment; this was corrected afterwards. The 2009 to 2010 audit reports had unqualified opinions.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Satisfactory
The project substantially contributed to better efficiency and sustainability of the urban transport system: the Government of Chile acted with a number of concrete reform measures that reduced transfers, enhanced network coverage, and increased operating cost coverage. Efficiency is rated substantial despite a two-year delay that was due to exogenous factors. 
Risk to Development Outcome:
Negligible to Low
Negligible to Low
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Satisfactory
The Government of Chile demonstrated a high level of expertise in dealing with the difficulties that arose with the launch of Transantiago as well as a high level of commitment to the reform agenda by carrying out studies with its own resources. 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The ICR has identified a number of lessons, of which the most important and relevant one is listed below with some adaptation:
  • Design of an appropriate results framework to measure outcomes under technical assistance loan is challenging. Not all projects fit into the same results framework template, which is better suited to the structure of infrastructure investments and projects with precise physical outcomes.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is concise and outcome-oriented. The lessons are evidence-based. It offers a thorough and detailed analysis of outcomes that were linked to the outputs produced under this technical assistance project (Annex 2). In the M&E section, however, the ICR could have discussed more the results framework as well as made an attempt to suggest better indicators to track outcome under this technical assistance, particularly as it brought up this issue in the lessons.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions