Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Secondary & Local Roads Project


  
1. Project Data:   
ICR Review Date Posted:
06/23/2014   
Country:
Georgia
PROJ ID:
P086277
Appraisal
Actual
Project Name:
Secondary & Local Roads Project
Project Costs(US $M)
 27.44  132.12
L/C Number:
C3938, L7671
Loan/Credit (US $M)
 20  90.69
Sector Board:
Transport
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  06/24/2004
 
 
Closing Date
10/31/2009 06/30/2012
Sector(s):
Roads and highways (96%), Central government administration (4%)
Theme(s):
Rural services and infrastructure (33% - P) Infrastructure services for private sector development (33% - P) Participation and civic engagement (17% - S) Decentralization (17% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Fang Xu
Victoria Alexeeva Christopher David Nelson IEGPS1

2. Project Objectives and Components:

a. Objectives:
The original project development objectives as in the Project Appraisal Document (p6-7) were " a) improve the economic and social well-being of the rural population in selected areas through upgrading of their secondary and local road networks; b) improve the effectiveness of the road department of the Ministry of Infrastructure and Development to maintain a cost effective and sustainable secondary and local road network; and c) improve the effectiveness of Road Department of the Ministry of Infrastructure and Development (RDMID) in its interaction with communities and its responsiveness to local needs".

The original project development objectives as in the Credit Agreement (Credit No. 3938 GE, Schedule 2) were " a) upgrade and rehabilitate secondary and local roads network; and b) strengthen RDMID's capacity to promote community participation in road management and manage road networks in a cost effective and sustainable manner".

The formally revised project development objectives as in the Loan Agreement (Loan No. 7671 GE, Schedule 2) for the Additional Financing "a) upgrade and rehabilitate the secondary and local roads networks; and b) increase RDMRDI's ( Roads Department of the Ministry of Infrastructure and Development) and local governments' capacity to manage the road network in a cost effective and sustainable manner ".

The formally revised project development objectives as as in the Restructuring and Additional Financing Project Paper (P4) were the same as the objectives in the Loan Agreement (Loan No. 7671 GE, Schedule 2).

This review assesses the original project development objectives formulated in the Credit Agreement (Credit No.3938) and the formally revised project development objectives formulated in the Loan Agreement (Loan No. 7671 GE).

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 03/19/2009

c. Components:
The original project had three components and an additional financing was approved on March 19, 2009. Based on the latest information provided by the project team, the project (the original and the AF) activities and costs are as follows:


1. Rehabilitation of secondary and local roads (total estimated cost US$117.54 million, total actual cost US$ 122.86 million).

  • The original component was to finance the rehabilitation of approximately 500-750 kms of paved secondary and local roads, including between 150 to 250 kms of local roads ( original estimated cost US$22.54 million, original actual cost US$25.82 million).
  • The additional financing was for the rehabilitation of about 450km of secondary and local roads, carrying out drainage improvements and providing signage and access to adjacent properties (AF estimated cost US$95.0 million, AF actual cost US$97.04 million).
  • During the first project restructuring on December 27, 2006, the scope of the original project component 1 was revised to "rehabilitation of about 250km of secondary and local roads, including about 75km of local roads" due to the higher construction cost and higher design standards utilized for the planned project activities.
  • During the third restructuring associated with the Additional Financing on February 25, 2010, the length of roads to be rehabilitated under the AF was increased from 450km to 590 to 630km ( ICR (p6))

2. Institution strengthening and capacity building (total estimated cost US$3.6 million, total actual cost US$4.46 million).
  • The original component was to finance the strengthening of the capacity of road sector institutions by improving engineering standards and data collection capabilities, establishing between four to six regional offices for network and works monitoring, development and implementation of regional road maintenance plans, improved interactions with local communities and traffic safety enhancement (estimated original cost US$2.4 million, actual original cost US$3.58 million).
  • The additional financing was for strengthening the capacity of road sector institutions including the capacity of the Roads Department of the Ministry of Infrastructure and Development and its six regional offices in management and maintenance of the secondary road network and the capacity of local governments in management and maintenance of the local road network (estimated AF cost US$1.2 million, actual AF cost US$0.88 million)

3. Design and supervision of the road works and to produce reports for the Mid-Term Review to evaluate the first year program (total estimated cost US$ 6.3 million, actual cost US$4.78 million).
  • The original component was to finance the design and supervision of the road works under the original project and to produce reports for the Mid-Term Review to evaluate the first year program (original estimated cost US$ 2.5 million, original actual cost US$ 4.78 million)
  • The AF was to finance the design and supervision of the road works under the AF (AF estimated cost US$3.8 million, AF actual cost US$4.15 million)

Along with the Additional Financing approved on March 19, 2009 was the second restructuring of the original project to revise the project development objectives to the one as formulated in the Loan Agreement (Loan No. 7671 GE). The revision of the development objectives reflected changes in the legal and institutional arrangement for the management and maintenance of the local roads in Georgia. Effective January 2007, the Roads Department of the Ministry of Infrastructure and Development (RDMRDI) was no longer directly involved in the management and maintenance of the local roads network, instead, the local self-government units are managing the local roads network (Additional Financing Project Paper, p4). As the project was supporting the improvement of both the secondary and the local roads network, it was important that the capacity building support was also extended to the local governments.

On October 28, 2011, a fourth project restructuring was approved to extended the project closing date from October 31, 2011 to June 30, 2012.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project cost: The original project cost at appraisal in 2004 was US$27.44 million, an additional financing of US$100 million (both the Bank and Borrower amounts combined) was provided in 2009 to scale up the project activities. The actual total project cost was US$132.12 million.

Financing: For the original project, Bank's financing was Special Drawing Rights (SDR) 13.8 million, US$ 20 million equivalent. For the additional financing, Bank's contribution was US$70 million. At project completion, US$0.31 million was cancelled. The actual Bank's financing was US$100.69 million ( due to exchange rate fluctuation of USD to SDR).

Borrower contribution: The Borrower contribution was US$37.14 at project appraisal, the actual Borrower contribution at project completion was US$41.42 million.

Dates: The original project closing date was October 31, 2009, with additional works under the Additional Financing, the closing date was extended by 24 months to October 31, 2011. The date was further extended to June 30, 2012 to allow the completion of two civil works contracts which had experienced some delays (ICR p7).


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
The original project development objectives were relevant with the Country's priority and Bank's Country Assistance Strategy for Georgia at project preparation. The Economic Development and Poverty Reduction Program of Georgia (EDPRP) finalized in June 2003 had two strategic objectives: rapid and sustainable economic growth,and a reduction in extreme poverty. The project development objectives aimed at improving the rural road transport system and enhancing the road transport system management capacity would contribute to the economic growth by reducing the transport cost for agriculture products and improving rural population's access to economic and social services, which in turn, would help generate more income for rural households and contribute to the economic growth in the country. Following the August 2008 conflict, the Bank reacted quickly by staying engaged and increasing Bank's support to the country, the additional financing of this project being a specific example. The original project development objectives were also relevant with the new Country Partnership Strategy (FY10-13), which sets two strategic pillars: meeting post-conflict and vulnerability needs, and restoring growth and competitiveness. More specifically, the Country Partnership Strategy called for focusing Bank's support on increasing connectivity via improved transport corridor and local roads network (Country Partnership Strategy p20). The relevance of original project development objectives is rated as Substantial.

The formally revised project development objectives were also relevant with the Country's and Bank's priorities both at project preparation and completion stages. As explained earlier, since the local governments are now responsible for the local road network management , RDMRDI is no longer directly interacting with the local communities, instead, it provides professional interaction, advice and guidance when necessary (Additional Financing Project Paper p4), the revision of the project development objectives reflected such institutional change. The essence of the original and revised objectives remained the same, which was to improve the rural roads network and aimed at reducing the transport costs and contributing to the economic growth and poverty reduction of the country, The relevance of the formally revised project development objectives is rated as Substantial.

b. Relevance of Design:
The original project activities covered two broad categories: the civil works for roads improvement and the institutional strengthening and capacity building. The result chain is clear and logical. The roads rehabilitation and upgrade will directly bring about improved roads network (the first sub-objective) and consequently increased accessibility and reduced transport costs; The institutional strengthening and capacity building will contribute to the improved roads network management (the second sub-objective). In addition, the project activities of improving road management agency's interaction with local communities such as promoting public participation in the development of five-year road program will also contribute to the achievement of the second sub-project development objective. The relevance of original project design is rated as Substantial.

The project design was still relevant to the revised project development objectives following the project restructuring and the additional financing. The result chain remained clear and logical. The roads rehabilitation and upgrade had been scaled up which would directly contribute to the achievement of the first sub-project development objective. A series of activities of strengthening the capacity of local government units were prepared and implemented following the change of institutional setup for local roads network management. These local government capacity enhancing activities (Component 3 of the additional financing) together with road sector institutions capacity strengthening activities (Component 2 of the original project and the additional financing) would help contribute to the achievement of the second sub-project development objective. The relevance of design to the revised project development objectives is rated as Substantial.


4. Achievement of Objectives (Efficacy) :


Achievement of the original project development objectives: a) upgrade and rehabilitate secondary and local roads network; and b) strengthen RDMID's capacity to promote community participation in road management and manage road networks in a cost effective and sustainable manner".

Outputs:
At project completion, the reported (ICR Annex 2) main project outputs were:
1) 693km secondary roads were rehabilitated with average travel speed of 50 kph.
2) 211km local roads were rehabilitated with all weather accessibility and with average travel speed of 30 kph.
3) 6 regional road management offices were established and operational with the function of data collection, maintenance planning and public participation.
4) 148 traffic police were trained;
5) 5-year rolling maintenance plan were developed with public participation;
6) Geometric design standards were adopted;
7) Guidebooks about the local roads maintenance for the local governments were produced;
8) A workshop to explain the use of guidebooks to the local governments was conducted;
10) The design of 900km roads was completed.

Outcomes:
The achievement of the first sub-objective of "upgrade and rehabilitate secondary and local roads network" could be evidenced by the length of secondary and local roads being upgraded and rehabilitated, and the impacts brought about by the roads improvement. The evidences are listed below:
1) 693km of secondary road were rehabilitated, meeting the target of rehabilitation of approximately 690-710 km of secondary roads;
2)The percentage of secondary roads in good or fair condition increased to 60 at project completion instead of 39 at project appraisal.
3) 211km of local roads were rehabilitated, exceeding the target of rehabilitation of of aproximately 150-170km of local roads;
4) The travel time on the project road setions was reduced by 19.9 percent (ICR p19);
5)13,431 person-month jobs were created as a direct result of project roads upgrade and rehabilitation (ICR p19).
6) The project beneficiary survey also identified that the local population in the project area now has improved access to social services and products&services centers. The supply of commodities and other products to the villages is more timely as well.

The achievement of this sub-objective is rated as Substantial.

The achievement of the second sub-objective of "strengthen RDMID's capacity to promote community participation in road management and manage road networks in a cost effective and sustainable manner" could be evidenced by the increased community participation in road management and more funds are being allocated for the roads management and improvement. The evidences are listed below:

1) The improvement of RDMID's road assets management capacity: the project team provided information that prior to the project, the costing of roads rehabilitation or improvement was input-based; the road design seldom took into consideration the traffic volumes and calculated the cost-benefit. In addition, very little attention was paid to the road maintenance. 70% of the roads network was not maintained in a proper manner and the roads condition was deteriorating. With the project's assistance, the approach of road assets management has changed, A GIS based inventory of roads with the information of the road has been created and the RDMID started to use HDM4 to monitor and plan the rehabilitation of the roads network, which would lead to huge cost saving both on the roads maintenance and rehabilitation/construction. The ICR (p16) reported that the unit cost of works carried out in the latter stage of the project based on the Manuals for the Design of Rehabilitation and Maintenance developed under the project was lower than that for the initial works under the project. The project helped institutionalize a cost effective road designs and maintenance approach.

2) The budget for the road maintenance was increased as monitored by total government expenditure. As reported by the ICR (data sheet), at project completion road maintenance and rehabilitation expenditure were Georgian Lari 184 million, exceeding the target of Georgian Lari 154 million.

3) 6 regional road management offices were established that would permit RDMID to acquire better local knowledge. At the same time, the local governments and communities could also voice better in the programming of road improvements.
4) The 5 year road maintenance plans were prepared using participatory approaches involving local governments and communities.

The achievement of this sub-objective is rated as "Substantial". In summary, all the sub-objectives of the original project development objectives have been substantially achieved.


Achievement of the revised project development objectives: "a) upgrade and rehabilitate the secondary and local roads networks; and b) increase RDMRDI's ( Roads Department of the Ministry of Infrastructure and Development) and local governments' capacity to manage the road network in a cost effective and sustainable manner ".

The first sub-objective of the revised PDOs is the same as the first sub-sector of the original PDOs, the achievement of this sub-objective is rated as Substantial.

The sub-objective of increasing RDMRDI's ( Roads Department of the Ministry of Infrastructure and Development) capacity to manage the road network in a cost effective and sustainable manner was Substantially achieved as evidenced by the information listed above.

After the responsibility of local road network was transferred to the local government, different modality of managing the local roads network was proposed, however, none was favored by the local government. The project team organized a workshop on application of micro enterprises, however, it did not yield expected uptake from the road sector. The project also provided some training to the local government; however, at project closing, there was no obvious evidence that the local governments' capacity of handing the local roads network has been substantially improved. The project team informed IEG that they were considering other modality of local road management such as Performance Based Contracting to be piloted in the Bank's subsequent operation in Georgia. The achievement of the sub-objective is Negligible.

In summary, the two sub-objectives of the revised project development objective have been substantially achieved, the third sub-objective was not achieved.


5. Efficiency:

The Project Appraisal Document (p11-12) reported that at the original project appraisal stage, economic analysis using HDM was carried out for the rehabilitation of about 22 secondary roads sections with the length of 1529km. In addition to the traditional economic analysis, the impact of the proposed road improvement works on poverty reduction was also assessed. Roads finally selected for were either because they were expected to generate higher economic rates of return or they were especially cost effective in serving poor populations. The poverty impact of the 10 roads sections in the project, measured as number of poor people per $1000 investment in road improvement was in the range of 10 poor persons/$1000 investment- 350 poor persons/$1000 investment. An ex-ante ERR for the overall program was not estimated, which, as the ICR accepts (p.30), is not a good practice because it does not allow an affective assessment of the robustness of the economic impact of the project.


At preparation of the additional financing, economic analysis was carried out using HDM for the 377km of the total 450km of secondary and local roads for which data was available. The average economic rate of return was 30.7% and net present value was US$114.6 million for the roads to be improved under the additional financing.

At project completion, an ex-post economic evaluation of the 49 project roads, totaling 844km was conducted with actual construction cost and latest traffic growth rates. The overall economic rate of return was 24.7%, lower than the overall rate estimated at the additional financing preparation stage. The main reasons for the lower actual rates of return were: 1) higher than estimated construction cost on some road sections under the original project as higher construction standards were adopted during the implementation; 2) traffic growth on some road sections was lower than projected due to the slow down of economy as a result of the crisis.

There was no significant cost and time overrun of the project. The total estimate cost for the original project and the AF was $127.44 million, the actual total project cost was $132.14 million. The planned project closing date was October 31, 2011 and the actual closing date was June 30, 2012. The cost overrun was mainly due to the increased in the Georgia consumer price index and improvement in the geometrical design standard (ICR p32) and the time overrun to allow the completion of two civil works contracts which had experienced some delays (ICR p7).

Overall, the project's funds were used efficiently under the original project and its additional financing. The efficiency of the project is rated as Substantial.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
Yes
24.7%
93%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The project outcome against the original PDOs is rated as Satisfactory given that both the project development objectives and the project design were relevant, the original project development objectives were substantially achieved and the project funds were used efficiently. The project outcome against the revised PDOs is rated as Moderately Satisfactory given that both the revised project development objectives and the project design were still relevant, two sub-objectives of the PDOs were substantially achieved and the project's funds were used efficiently, however, very little has been achieved under the third sub-objective of "increasing the local governments' capacity to manage road works in a cost effective and sustainable manner ". The disbursement rate was 23% before the approval of the revised PDOs and 77% after the approval of the revised PDOs. The weighted outcome of the project is Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The main risk to development outcome remains the local government's capacity of managing the local roads network since the local government was still trying to figure out the right modality of managing the local roads network. The Bank will try Performance Based Maintenance modality under the on-going Secondary and Local Roads Project 2, the effectiveness of the Performance Based Maintenance Modality is to be tested. The secondary roads network management is likely to be sustained as RDMID with increased capacity and increased road maintenance budget is responsible for managing it. The overall risk to development outcome is moderate.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:
The project focused on the secondary and local roads improvement, which was considered as the priority of the road sector as the secondary and local roads are providing important access for the rural households to the economic and social services. The project activities included both the physical investment and the institutional strengthening and the selection of project activities was based on the careful consideration of both economic and social impacts. The ICR notes (p.23) that an analysis of the local domestic industry could have been usefully included in the project design to provide a more strategic basis for efforts to increase the capacity of the industry.

Quality-at-Entry Rating: Satisfactory

b. Quality of supervision:
The ICR(p23-24) reported that the Bank provided timely guidance during project implementation. The number of restructuring that the project went through and the additional financing provided reflected the Bank team's productivity in managing the project and supporting the Borrower in implementing the project. However, the ICR (p24) reported that there was no road engineer in the team for the first 3-4 years of project implementation, which limited the outputs from site visits and the team's ability to carry out its due diligence responsibility. Yet, the additional information provided by the country team showed that there was always road engineer as part of the team.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The Government demonstrated strong commitment to the project by increasing substantially the roads maintenance budget allocation, albeit with the crisis in the country. The Government also increased its contribution to the project so as to ensure that the targeted roads' improvement could be fully achieved.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:
The implementing agency refers to the Roads Department of the Ministry of Regional Development and Infrastructure (RDMRDI).The RDMRDI completed the secondary and local roads improvement as planned under the project with very small cost increase and time extension. The ICR (p24) also reported that during project implemention, the RDMRDI was willing to change its practices and adopt new methods and standards such as using the new Manual for the Design for Rehabilitation and Maintenance to reduce the unit cost for road works. Such change was not easy given the long tradition of applying Soviet era practices. The ICR notes that there were some shortcomings in the quality of works that were partly attributed to inadequate supervision of the works by the supervision consultants and infrequent monitoring by the RDMID of the performance of the consultants and contractors. It was agreed that these shortcomings could be financed from contingencies under the contracts (p.13).

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The M&E framework of the original project had indicators to capture the project outputs and reflect the outcomes. The M&E framework also included the indicators to reflect the possible impacts that the project may bring about such as job created by the project activities. The M&E framework was updated to accommodate the change of the project development objectives at additional financing stage. However, there were limited indicators to measure the achievement of the revised objective of "improving local government's capacity to manage the road network in a cost-effective and sustainable manner".

b. M&E Implementation:
Project progress reports were prepared regularly and the information on the indicators at all levels was collected and reported. All information was used as a tool to monitor and evaluate the project implementation progress and the achievement of the objectives.

a. M&E Utilization:
The M&E was used to measure the project progress as evidenced by the citation of M&E information in the Aide Memoires and Implementation Status Reports. The M&E was also used to adjust the scope of the planned road rehabilitation to reflect the changes in approach to the road rehabilitation and increase in the contribution of the Government.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:
Environmental: The environmental category for this project is "B". The Bank safeguards policy triggered was OP/BP4.01. An Environmental Management Framework was prepared at appraisal and specific Environmental Management Plan was completed for each road package during project implementation. The environmental category remained as "B" with the additional financing at expanded scope of road works, The ICR (p15) reported that the project was overall in compliance with the Bank environmental policy throughout the implementation, however, there were noted shortfalls during project implementation such as insufficiency of signing of work sites and disposal of waste from road sides, those shortfalls were later on corrected.

Social: The road works did not expect to involve any land acquisition at project preparation. During project implementation, a minor land acquisition took place which was caused by the necessity of realigning a road section due to the erosion of the river bank. However, the Bank's Involuntary Resettlement Policy was not triggered due to the small and singular nature of the occurrence.

b. Fiduciary Compliance:
Procurement: The ICR (p15) reported that the procurement was relatively well handled, there was not reported non-compliance of the Bank procurement policy and process. The ICR also reported that as the Borrower became familiar with and appreciated the merit of the Bank procurement process, they decided to use the Bank procurement methods for activities funded by the government budget.

Financial Management: The ICR (p16) reported that the financial management of the project was overall satisfactory except a few late submission of the financial monitoring reports and late payments. There was no reported ineligible expenditures or non-compliance of Bank fiduciary policies.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
Under the original PDOs, relevance of objectives and design, as well as efficacy and efficiency were substantial. For the revised PDOs, two out of three sub-objectives were substantially achieved and the project's funds were utilized efficiently. However, very little has been achieved under the new sub-objective of "improving the local government's road works management capacity ", therefore, the outcome was rated as Moderately Satisfactory.  
Risk to Development Outcome:
Negligible to Low
Moderate
Under the revised project development objective, very little has been achieved on improved local government capacity. There is an ongoing follow-up Bank's operation in this area, however the impact is yet to be seen. The risk is considered as moderate.  
Bank Performance:
Satisfactory
Satisfactory
 
Borrower Performance:
Satisfactory
Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The ICR generated five lessons, two of them were general and applicable to all the project implementation; the ICRR selected three lessons, which are more specific:
1. For projects involving the rehabilitation of roads scattered over several locations, it is prudent to trigger OP/BP 2.12 on Involuntary Resettlement and prepare an Resettlement Management Framework. This would then allow the project team deal with any unexpected need to acquire land for project purposes.
3. An assessment of the local construction industry should be carried out during the preparation of a project with significant civil works activities so as to facilitate the involvement of the local construction industry.
5. The Bank should assess the implementing agency's capacity before loading the same agency with more new projects.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR provided comprehensive information on the project implementation and reported the project outputs and outcomes. However, the ICR could have been more concise by avoiding some repetitions, for example, the same text on the community consultations appeared twice in the ICR both at the social safeguard section (p11) and the project design section (p9-10). The ICR did not provide estimated and actual cost information at project component level (Annex 1, table (a)). In addition, the ICR information on the Bank team composition was not accurate either.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions