Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Road Maintenance & Rehab 3


  
1. Project Data:   
ICR Review Date Posted:
12/29/2013   
Country:
Poland
PROJ ID:
P096214
Appraisal
Actual
Project Name:
Road Maintenance & Rehab 3
Project Costs(US $M)
 376.2  452.39
L/C Number:
L7384
Loan/Credit (US $M)
 180.2  194.45
Sector Board:
Transport
Cofinancing (US $M)
 166  202.77
Cofinanciers:
Board Approval Date
  06/06/2006
 
 
Closing Date
09/15/2011 09/15/2011
Sector(s):
Roads and highways (93%), Central government administration (7%)
Theme(s):
Infrastructure services for private sector development (50% - P) Injuries and non-communicable diseases (25% - S) Administrative and civil service reform (25% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Midori Makino
Peter Nigel Freeman Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
Project objective, stated in the Loan Agreement was to “assist the Borrower in continuing improving the effectiveness of the country’s national road rehabilitation and maintenance systems, with emphasis on quality, efficiency, financial viability, and road user satisfaction”. The objective in the PAD was to continue improving the effectiveness of Poland's national road rehabilitation and maintenance systems, with emphasis on quality, efficiency, financial viability, safety, and road user satisfaction. The objective from the PAD is used for the purpose of this ICR review because it explicitly includes the added dimension of road safety for which relevant components and indicators were included in the project design.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
The project had four components.

(a) Road Maintenance and Rehabilitation Program (Appraisal: US$167.0 million, Actual: US$181.92 million)
This first component followed the same procedure as in the previous two World Bank supported projects in 2004 and 2005 and pooled resources provided by the Bank and European Investment Bank to reduce the existing maintenance backlog, based on GDDKiA's budget for 2006 approved by the Parliament in February 2006. The component would finance (i) a total of 165 contracts for physical works for periodic maintenance and rehabilitation throughout the National Road Network; (ii) at least one pilot contract for Performance-based Management and Maintenance of Roads; and (iii) a public awareness campaign in support of the Government's road construction and rehabilitation program, based on the communication strategy developed by GDDKiA in 2005. The Bank would contribute about Euro 139 (48 percent) to this component.

(b) GDDKiA Management Information System (Appraisal: US$3.0 million, Actual: US$ 4.4 million)
This second component would complete the implementation of GDDKiA's management information system which was initiated under the previous project. Specifically, the project would implement two packages including Wide Area Network services, and hardware and software of a security system for the protection of Wide Area Network, Management Information System, and Internet Access. This component would be fully financed by the Bank.

(c) Road Safety (Appraisal: US$6.6 million, Actual: US$6.85 million)
This third component would support road safety campaigns based on themes such as "alcohol and driving", "speeding", and "safety of children" and "pedestrian safety", as well as technical assistance to the Secretariat of the National Road Safety Council. This component would be fully financed by the Bank and contribute to the package of road safety initiatives co-financed also by the EU and EIB.

(d) Technical Assistance to Ministry of Transport and Construction (Appraisal: US$ 3.6 million, Actual: US$ 1.28 million)
This component would support (i) an Intelligent Transport System study aimed at establishing national standards for gathering and transferring information; (ii) Technical assistance to help better and faster absorption of EU funds - output of such assistance would be a Handbook for the use by Ministry staff as well as for project beneficiaries; (iii) assistance to improve road technical standards and traffic management; and (iv) audit of the road maintenance and rehabilitation projects. The Bank would finance Euro 3 million (78 percent) of this component.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
The project was approved on June 6, 2006 and closed on September 15, 2011 according to the original schedule. Actual project cost was $452.39 million versus the planned total cost of $376.2 million. The Bank's contribution to the project decreased from the original estimate of 48 percent to 43 percent as the proportion of the funding from the Borrower ($30 million or 8 percent of total to $55 million or 12 percent of total) had increased. With the exception of the technical assistance to the Ministry of Transport and Construction, which disbursed only about a third of the amount planned, all of the other components exceeded the planned budget. The increase was due to the appreciation of the Euro, in which the loan was denominated, versus the USD. In the case of the physical works the cost increases were also due to reallocation of funds to road works from other components and an expansion of the works undertaken.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
The relevance of the project objectives was High. The Country Assistance Strategy of 2002 prepared by the World Bank at the time of Poland’s accession to the EU, recognized the importance of strengthening road sector institutions to enable absorption of the vastly expanded resources to be provided by the EU and other European institutions. The Strategy also noted that the incidence of unsafe and congested roads and high transport costs added to regional disparities and made the country less attractive to private investment. The project objectives reflected these challenges that the country was facing at that time and established an appropriate role for the Bank’s comparative strengths. The Country Partnership Strategy of 2009 observed that while the EU and other European institutions, notably the EIB were the main partners and external sources of financing for Poland's road sector, the Bank could continue to play an important role in helping to build institutions and in complementary investments.

b. Relevance of Design:
The relevance of the project design with regard to its approach and the components financed to achieve the objectives was Modest. The hybrid approach used in the project design was the same as that used for the previous two road maintenance and rehabilitation projects. This approach remained as relevant as before: Poland urgently needed access to quick disbursing road maintenance and rehabilitation funds since the counterpart funding requirement of the EU program was draining resources from the road sector. Therefore, it was critical to use an approach, such as the SWAp that was designed to facilitate fast disbursement of project funds to stabilize the situation. The road maintenance and rehabilitation component that supported the civil works was appropriate for achieving the quality of Poland’s national road network and consequently road user satisfaction, and the technical assistance would contribute to the efficiency of the systems. Poland also needed to continue improving its road safety record as an EU member, to reduce road accident fatality rates, and the Bank funded road safety component was critical to this effort. The project objective as specified in the financing agreement, however omitted the reference to road safety while it was included in the PAD objective, component, and indicators. While the project’s design was appropriate and relevant in these respects, it did not address the sub-objective of achieving financial viability through stable and reliable funding for road maintenance. This was critical to the sustainability of the project and the road sector, and its absence from the project design and monitoring indicators proved to be a major shortcoming.


4. Achievement of Objectives (Efficacy) :

The overarching objective of the program was to "continue improving the effectiveness of Poland’s national road rehabilitation and maintenance systems, with emphasis on quality, efficiency, financial viability, safety and road user satisfaction'. Assessment of each of the five sub-objectives; quality, efficiency, financial viability, safety, and road user satisfaction are presented below.

1. Quality

Outputs: The project implemented 218 contracts with a total length of 1,103 km which corresponds to six percent of the national road network.

Outcomes: Between 2003 and 2009, the percentage of national roads in good condition increased from 37 percent in 2003 (baseline of the first project) to 59.6 percent in 2009, essentially achieving the project target of 60 percent in good condition. In 2012 the percentage of roads in good condition increased to 62.7 percent as a result of increased periodic maintenance funding. The load bearing capacity of the road network improved significantly beyond the target of 3,300 km from the baseline of 2,191 km to 5,897 km.

The Bank has financed a significant portion of the civil works. An IEG mission which was undertaken after project closure learnt that the works undertaken had been subjected to technical audits and found to be of good quality. The achievement of the quality sub-objective was therefore rated as High.

2. Efficiency

Outputs: The pilot for the performance based maintenance and management of roads was not implemented as scheduled because of the strong resistance from GDDKiA operating staff, deficiencies in the quality of the consultant and frequent changes of GDDKiA management. However, towards the end of the project GDDKiA did procure a four year maintenance contract termed "sustained standards" contract, similar to what was originally prepared under the project, for 84 km of the expressway and planned to use this contract for all newly constructed highways and upgraded roads. In addition some routine maintenance works were outsourced. As of 2013, the government has rolled out this new form of performance based contract in at least six road networks.

With regards to the modernization action plan, the project supported the reorganization of GDDiKA into three departments; (i) planning; (ii) investment; and (iii) maintenance and establishment of units for traffic management and quality control of construction. Functional reclassification of the road system however was not implemented as scheduled because of political resistance to reallocate the budget. Some progress has been made to revise the procurement rule, and to modernize human resources, information and data, and financial management. However, the use of the system is limited to accounting and financial management.

The enabling environment to use the HDM-4 system was created through system installation and capacity building of GDDKiA staff, and the system is being used for the purpose of assessing each investment by all sixteen regions. In at least two to three regions, the system has also been used for optimizing the allocation of resources at the network level under budget constraints. Some regions are experiencing difficulties using the system for the network analysis and prioritization purpose because of the complexity in collecting the necessary data.

Outcomes: The sustained standards contracts implemented by GDDKiA found that using private contractors, the same maintenance work could be carried out with only 37 percent of the costs, saving 63 percent of the maintenance budget. According to the TTL, the government continues to approach the World Bank to seek for more guidance in handling performance based contracts. Albeit with delays, the project at least partly contributed to the government's plan to use performance based road maintenance and management contracts. GDDKiA has also managed to contain its number of employees despite the increasing budget and this has resulted in significant productivity increase for the agency. There is limited evidence that demonstrate successful efficiency outcomes of GDDiKA’s modernization action plans, but the increasing use of HDM-4 system by the regional offices of GDDKiA is also likely to contribute to the efficiency of the sector.

The efficiency sub-objective of the project is therefore considered to be partially achieved, and therefore rated as Substantial.

3. Financial Viability

Outputs: The project did not include any components or features that would support this sub-objective.

Outcomes: The National Road Fund which would have provided a stable source of maintenance funding, has been specifically instructed to fund only new construction with minor exceptions for works justified on road safety grounds. Counterpart financing requirement for accessing EU funds strained the Government’s financial resources available for the road sector. The severity and length of the financial crisis that has impacted the Government’s finances since 2009 has exacerbated the situation. As a result, the funding for rehabilitation and maintenance was reduced in 2010 and 2011, after the closing of the project. According to the TTL, this the government has recently increased its budget for periodic maintenance but it is unclear whether this increase in allocation of maintenance funding will be sustained during the next few years.

While the routine road maintenance funding is consistently being funded, securing sufficient amount of periodic maintenance funding in a reliable manner continues to be a challenge. Therefore the achievement of the financial viability objective is rated as Modest.

4. Road Safety

Outputs: Outputs related to road safety included; (i) individual and institutional consultancy services to the National Road Safety Council; (ii) road safety studies and analysis, (iii) soft measures in the form of public relations activities and awareness campaigns; and (iv) support for investments and implementation of safety equipment and facilities. Large road safety campaigns were launched to modify behavior to promote road safety, including topics such as “driving vehicles under alcoholic influence”, “driving vehicles at unreasonable speed”, and”usage of seat belts and child restraints”. A blackspot elimination program on regional roads were also implemented.

Outcomes: The number of road accident fatalities in Poland declined from 5,500 (Baseline of the first project) in 2003 to 3907 in 2010 which was well below the project target of <4700 fatalities. However, Poland failed to achieve its road safety program's Vision Zero target of reducing the fatalities by 50 percent between 2003 and 2013, and continued to have the highest fatality rate in the EU at 110 per million population, compared to the EU average of 60 in 2012. According to a report produced by Global Road Safety Facility in 2013, EU average in reducing road deaths over the period 2001-2011 was 45%, compared to 24% in Poland. Achievement of the road safety objective is therefore rated as Modest.

5. Road User Satisfaction

Outputs: After the baseline survey conducted in 2004, subsequent surveys were undertaken in 2005, 2009 and most recently in 2011. The last two surveys were carried out during the project implementation period.

Outcomes: The 2011 survey found significant improvements in road user perceptions of both GDDKiA as an institution as well as of the physical condition of the national roads network. In 2005 only 2-3 percent of users were found to have a positive opinion of the national road network, whereas by 2009 this proportion had increased to 30-35 percent, and in 2011 it was at 38 percent. While the results of the surveys show the desired upward trend, their frequency and regularity could be improved. Achievement of this sub-objective is, nevertheless, rated Substantial.


5. Efficiency:

95 percent of the 2005-2006 contracts and 17 percent of the 2008 and 2011 contracts contributed to an ex-ante economic rate of return (ERR), using the HDM-4 model. The overall ex-ante ERR was 37 percent and the median ERR for periodic maintenance for overlay of 30-50mm was 43 percent, for rehabilitation it was 38 percent, and for widening of roads to 7.0 meters it was a 28 percent. improvement An ex-post economic evaluation was conducted for the Implementation Completion and Results Report based on the actual traffic growth rate (3.4 percent p.a. versus 5.2 percent p.a. assumed at appraisal) and the actual costs (which were 2 percent above estimated costs). The rate of return for a sample project was 33.4 percent, indicating a small reduction but still yielding an acceptable ERR. GDDKiA’s assessments point to 75 percent of contracts having an ex-ante ERR above 20 percent. While the ex-post analysis was not done for all roads, the result of the sample project analysis indicates that it is likely that most roads had a rate of return above the benchmark of 12 percent and close to the appraisal value of 37 percent.

Efficiency for this project is therefore rated as Substantial.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
37%
89%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Relevance of the objective was high, relevance of design was modest, and efficiency was substantial. Achievement of the quality objective was high, efficiency and road user satisfaction were substantial, and financial viability and road safety were modest. Based on the above assessments the overall outcome rating is Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

Risk to development outcome is rated Significant. Maintaining a balance between rehabilitation needs of the existing national roads network and new construction and improvement works, was an important consideration in the Bank’s support for Poland’s road maintenance and rehabilitation program. Government commitment to rehabilitation of the network, and to maintaining the network in a stable condition at around 60 percent in good condition, appears to have been sacrificed in recent years in order to support EU funded new construction programs. Depriving road maintenance funding of its allocations from the Road Fund is one of the more unfortunate consequences of this shift in policy. Whatever the merits or imperatives driving that policy, it is not in line with the development objectives of the program. Also, while the fiscal constraints facing the Government since the financial crisis in 2009 may be the reason, there appears to be an inadequate appreciation of the damage done both to the physical network itself, and to the institutional capability for maintenance, by the frequent neglect of maintenance. Tolling of the newer expressways now under construction may provide additional revenues to the Road Fund but the Government needs to give a high priority to restoring the allocations for road maintenance.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:
The project concept with regard to objectives and the design of the components was certainly of high relevance to Poland’s needs at the time and continues to be so even now. The strong focus on road safety, systematic road maintenance planning using HDM-4, and reforms of road sector institutions indicate a high degree of Bank staff commitment to road sector development. However, in the execution of the project and in its monitoring and evaluation framework, insufficient attention appears to have been given to ensuring stable funding for maintenance of the network. The financial viability lies at the core of effective road network management, and not including a monitoring indicator on this has proved to be a serious deficiency in the design of the project.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:
While the number and staffing of supervision missions was adequate, their effectiveness could have been greater. The Bank should have reacted more forcefully on the Government’s refusal to utilize the National Road Fund for maintenance and rehabilitation of the network. It is understandable that owing to the fast disbursing nature of the three operations, the Bank had little leverage over Government’s actions subsequent to the financial crisis in 2008/9. However, it would have been worthwhile to take a strong position, by for instance suspending disbursements or other such action, to indicate to Government the seriousness with which the Bank views such developments. Despite the shortcomings the Bank support has contributed to the road maintenance and rehabilitation program in Poland, as evidenced in their continued request for the Bank’s technical assistance even after project closure

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The Ministry of Infrastructure/Transport has performed satisfactorily with regard to rapid compliance with the conditions of Board presentation and effectiveness, as well as complying with its obligations regarding financial management and reporting. However, the Government’s commitment to the objective of improving the condition of the national road network appears to have been weakened by the fiscal crisis faced by Poland since 2008/9. The prohibition on the use of the National Road Fund for maintenance expenditures, which should correctly be its only valid use, was caused by a shift in the government’s short term priorities during the fiscal crisis, and now that the situation has improved the prohibition could be lifted to restore funding of the maintenance and rehabilitation program. With regards to road safety, the National Road Safety Council and other agencies were not given clear leadership, accountability, and responsibility to ensure sound coordination between and among the stakeholders.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:
GDDKiA performed well on its responsibilities under the project with a few shortcomings. GDDKiA implemented the road maintenance and rehabilitation component effectively, disbursing a bulk of the funds within a little more than a year, and the works produced have been found to be of good quality. The target to be achieved for road network condition was essentially met. Though the performance-based management and maintenance of roads pilot contract was unsuccessful, based on this experience GDDKiA did succeed in introducing a modified form of performance based maintenance contract (Sustained Standards contract) that was better suited to conditions in Poland and created the possibility of progressive outsourcing of road maintenance works. Implementation of the Management Information Systems technical assistance components did suffer long delays and was only partially successful. The Enterprise Resource Planning system in particular encountered significant problems and some modules had to be dropped. Likewise the Intelligent Transport System Study was dropped due to lack of coordination between GDDKiA and the consultant on the objectives of the study. National Road Safety Council performed well on its campaigns to boost public awareness in a number of areas such as seat belt use, alcohol and driving, speeding, child safety, and pedestrian safety. However, as mentioned above the enabling environment was not established for it to meet the national targets for road safety.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
M&E Design: The M&E system designed for the project essentially retained the arrangements made for the first and second projects. The indicators covered most of the key objectives – quality, efficiency, safety and road user satisfaction, the one exception being financial viability. With regard to the condition of the road network, it relied on the regular data collection carried out by GDDKiA’s Department of Studies. The other aspect of M&E is the road safety statistics on road accidents and fatality rates. These data were collected by the National Road Safety Council and they have maintained detailed data on accidents, injuries, fatalities, and vehicle fleet as necessary to support the M&E design of the project. That the M&E indicators did not address sub-objective of ensuring the financial viability of the road sector, in particular for the funding of maintenance, was a critical oversight. The shortfalls in maintenance funding now appear to have jeopardized the sustainability of the improvements made under the program.

b. M&E Implementation:
M&E Implementation: GDDKiA has a comprehensive data collection system for road conditions and traffic levels as needed to support the monitoring indicators in these aspects. The National Road Safety Council maintains up-to-date information on traffic and accidents statistics. Therefore data on road pavement conditions and safety statistics were well provided for in the M&E framework. Road user perceptions were the subject of four surveys undertaken in 2005, 2006, 2009 and 2011 and these were useful in recording improvements in user perceptions of the road network. As noted earlier, the main deficiency in the M&E framework was the absence of any indicators or data on the financial and budget situation for the maintenance of the road network. Presently, in addition to the fuel levies being deposited in the Road Fund, there are toll revenues being generated from the new motorway developments. However, the M&E framework for the project does not produce a consolidated view of these resources or their availability to fund maintenance.

a. M&E Utilization:
M&E Utilization: Both systems established for monitoring and evaluation, the HDM 4 based road data and the collection of road safety statistics, continued to perform satisfactorily, and the data produced has assisted the Government in judging the effectiveness of both the road rehabilitation and the road safety programs.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
The project’s works triggered only minor environmental impacts since these were small scale rehabilitation works within the existing road’s right of way and did not require any land acquisition or entail any displacement of people. The Polish environmental safeguards upon which the project relied had been reviewed and found to be adequate to the Bank’s requirements.

b. Fiduciary Compliance:
Annual procurement post reviews confirmed that procurement activities related to the works component were implemented using country procurement systems, and were carried out satisfactorily. Financial management was rated “moderately satisfactory” by Bank supervision missions principally because the implementation of the SAP financial systems was delayed. In general Financial Management/Disbursement processes have been satisfactory with clean audit opinions. The Audit reports and the Financial Management Reports were generally received on time.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Significant
Significant
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
A combination of the key lessons learnt based on the ICR and on IEG's review are summarized below:

ii. The use of country systems, combined with adequate ex-post audits can contribute to fast disbursement of quality works. The Sector Wide Approach adopted for the implementation of civil works using the borrower’s own systems worked well in a country where there is already a functioning system. Fiduciary and safeguards oversight can be based on ex-post audits of the efficacy of internal controls. Quality control based on third party technical audits also proved adequate and efficient.

ii. Objectives are more likely to be achieved when they have comprehensive monitoring indicators. While proper indicators were set for most objectives, there were no indicators linked to the objective of achieving financial viability. This may have contributed to the program’s failure to achieve progress on this sub-objective.

iii. It is crucial to take into account the budget constraints when using HDM-4 analysis in order to complete the rehabilitation program within a finite number of years. Using HDM-4 to analyze only the rates of return on rehabilitation projects ignores the financial viability of the program, and this does not eliminate the maintenance backlog. The rehabilitation program should be designed to deliver a maintainable network within a reasonable time frame, and this requires taking into consideration a realistic budget envelope over the program period.

iv. It is important to ensure that sufficient disbursement leverage is retained to ensure the satisfactory implementation of the institutional development components, as well as a realistic time-frame for the operation. This was the third project and the disbursements were fast but the Bank experienced difficulty leveraging over the policy actions and institutional development programs which took longer to mature.


14. Assessment Recommended?

Yes
Why?
The assessment has already been carried out with the first two Road Maintenance and Rehabilitation Projects, to feed into IEG's transport sector evaluation, delivered in FY13.

15. Comments on Quality of ICR:

The ICR was concise, ensured internal consistency, and generally followed the ICR guidelines. In carrying out the assessments, the ICR used relevant statistical data and other sources of information as evidence. The efficiency analysis has a thorough analysis of the ERR for the project but there are no statements about the project components for technical assistance activities, other than for road safety.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions