Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Road Maintenance And Rehabilitation Project


  
1. Project Data:   
ICR Review Date Posted:
10/28/2013   
Country:
Papua New Guinea
PROJ ID:
P004397
Appraisal
Actual
Project Name:
Road Maintenance And Rehabilitation Project
Project Costs(US $M)
 59.18  112.65
L/C Number:
C4298, L7119
Loan/Credit (US $M)
 40  77.5
Sector Board:
Transport
Cofinancing (US $M)
 0  15.95
Cofinanciers:
Board Approval Date
  06/06/2002
 
 
Closing Date
12/31/2006 06/30/2012
Sector(s):
Roads and highways (80%), Central government administration (11%), Sub-national government administration (9%)
Theme(s):
Rural services and infrastructure (25% - P) Infrastructure services for private sector development (25% - P) Decentralization (24% - P) Participation and civic engagement (13% - S) Micro Small and Medium Enterprise support (13% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Fang Xu
Roy Gilbert Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
“To assist the Borrower in promoting an efficient, safe and reliable roads transport system in the participating provinces through: (a) the improvement of reliable roads transport system in the participating provinces through: (a) the improvement of selected road segments; (b) strengthening strategic planning and management of the road sector; and (c) strengthening the institutional arrangements for road maintenance, including private sector participation.” (Loan Agreement, Schedule 2, and PAD p. 2)

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
With the Additional Financing (AF) of US$37.3 million approved on April 26, 2007, the original project components and the additional activities were as follows:

1.Road maintenance and rehabilitation of national roads (total estimated cost US$55.42 million, total actual cost US$ 76.48 million). The original component was to finance annual routine maintenance on about 2,200km and restoration of about 440km of national roads (original estimated cost US$26.70 million, original actual cost US$35.62 million). The additional financing (AF) was for the routine maintenance of about 200km national roads and the rehabilitation of about 200km national (AF estimated cost US$28.72 million, AF actual cost US$40.86 million).

2.Road maintenance and rehabilitation of provincial roads (total estimated cost US$10.52 million, total actual cost US$12.41 million ). The original component was to finance routine maintenance on about 900km and restoration of about 200km of provincial roads in six provinces (original estimated cost US$7.55 million, original actual cost US$9.64 million). The AF was to finance routine maintenance of about 50km and the restoration of about 20km of provincial roads (AF estimated cost US$2.97 million, AF actual cost US$2.77 million).

3. National bridges (total estimated cost US$16.46 million, total actual cost US$ 9.19 million). The original component was to finance the maintenance, rehabilitation and replacement of national bridges (original estimated cost US$8.7 million, original actual cost US$ 5.12 million). The AF was to finance the maintenance or rehabilitation of about 30 national bridges (AF estimated cost of US$7.76 million, AF actual cost was US$4.07 million).

4. Provincial bridges (total estimated cost US$5.67 million, total actual cost US$ 0.52 million). This component was to finance the maintenance, rehabilitation and replacement of bridges on provincial roads. There was no AF for this component.

5. Implementation support (total estimated cost US$13.99 million, total actual cost US$11.1 million). This component was to finance the consulting services and equipment to support project implementation (original estimated cost US$7.88 million, original actual cost US$5.39 million). The AF was for further implementation support (AF estimated cost US$6.11 million, AF actual cost US$5.71 million).

6. Road sector support (total estimated cost US$2.24 million, total actual cost US$2.95 million). This component was to finance improvement of institutional capacity of Department of Works and the capacity of small-medium sized contractor for road maintenance operations (original estimated cost US$2.29 million, original actual cost US$1.93 million). The AF provided further road sector support (AF estimated cost US$1.01 million, AF actual cost US$1.02 million).

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Cost: The original estimated project cost was US$59.18 million. In 2007, an additional financing (both the Bank and Borrower amounts combined) of US$53.95 million was provided. The actual total project cost was US$112.65 million.

Project Financing: The Bank approved US$40 million IBRD loan for the original project with US$39.97 million disbursed; and US$37.3 million IDA credit for the additional financing with US$24.24 million disbursed at project completion.

Borrower Contribution: The original Borrower contribution commitment was for US$25.07 million, Under the additional financing, the Borrower contribution commitment was for US$21.6 million. The actual total contribution from the Borrower was US$35.13 million.

Project Closing Date: The original project closing date was December 31, 2006 and the actual closing date was June 30, 2012. There were four extensions of the project closing date, two of them were for the original project, which was a loan and two extensions were for the additional financing, which was a credit. Details are as follows:

  • The first extension was for 12 months to December 31, 2007 to compensate for earlier delays in project implementation as a result of slow counterpart funding provision, lack of familiarity with the Bank procedures and slow response from some participating provinces;
  • The second extension was for 24 months from December 31, 2007 to December 31, 2009 when AF of US$37.3 million was approved with a closing date of December 31, 2009.
  • The third extension was for 24 months from December 31, 2009 to December 31, 2011 to accommodate delays of project implementation due to the longer than expected time spent to get the National Executive Council's approval of new Employer's Project Manager contract for the project and the lengthy visa approval time for the consultant.
  • The fourth extension was 6 months from December 31, 2011 to June 30, 2012 in order for the Government to disburse all the available bank credit funds.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Rated: Substantial
The project objectives were relevant to the Government strategy and Bank's Country Assistance Strategy (CAS 19590-PNG) for Papua New Guinea at the project preparation stage. In the Government Medium-Term Development Strategy (1997-2002, “A Bridge Into the 21st Century”), the Government called for a stronger role of local governments in the provision of public services (PAD p.5). The sector-related CAS (19590-PNG ) objective also focused on "improving the quality and reliability of the country's infrastructure, particularly the provincial and rural road network, to allow for more cost-effective movement of goods and services and to promote economic development in rural areas".

The project objectives remain relevant to the Bank's more recent FY08-11 CAS which identified that one of seven pro-poor expenditure priorities is "rehabilitation and maintenance of transport infrastructure", which is fully supported by the project. They also remain relevant to the Papua New Guinea Medium-Term Development Plan 2011-2015 that sets a goal of having 25,000km of national roads in good condition by 2030. This plan also states that there is need to provide more funding to the sector to encourage private sector participation to achieve that goal (ICR p15).

b. Relevance of Design:
Rated: Substantial
The project design focused on three kinds of actions: i) physical investments in selected national and provincial roads to bring them up to good condition; ii) institutional capacity building and management skill improvement in roads management agencies; and, iii) interventions to enhance the contracting industry and promoting the private sector locally. The results chain following these actions was straightforward, logical and well understood by the project designers. In different ways, each of the three kinds of actions would directly contribute to making the road systems more efficient, reliable and safer. These actions would help enable faster and smoother traffic flows; the transport system would be managed more effectively by the road management agencies. In these ways the project development objectives of "promoting an efficient and reliable roads transport system in the participating provinces" would be achieved. Road safety in PNG was undermined by a series of factors including drunken driving, inadequate road safety awareness by drivers and pedestrians, lack of uniform safety standards for traffic engineering and accident black spots, and inadequate enforcement of traffic rules. The project had no stand alone "road safety" component, but it did include activities such as road condition improvements, introducing uniform and engineering standards, and procurement of traffic counting equipment to provide fundamental data on road user usage to inform maintenance priorities, engineering designs and road safety considerations. But the lack of an explicit road safety activity targeting at road safety behavior and awareness was a shortcoming of the project design, that lessened the project's direct contribution to promoting the road safety.


4. Achievement of Objectives (Efficacy) :

Outputs:
As reported by the ICR (Annex 2), the main project outputs are as follows:
i) 600km of national roads were restored, short of the original target of 640km;
ii) 447km per annum of national roads were provided with routine maintenance, short of the original target of 750km (revised target of 543km)
iii) 166km per annum of provincial roads were provided with routine maintenance, exceeding the original target of 50km per annum.
iv)174km of provincial roads were restored and maintained short of the original target of 220km;
v) 46 national bridges were rehabilitated and maintained short of the original target of 65 bridges.
vi)79 small and medium domestic contractors were trained;
vii) 45 Department of Works central and provincial level staff trained in six provinces;
viii) Bridge Inventory and bridge management systems were established for sustainable planning and management of bridge assets.

Outcomes
The achievement of the project objective is assessed by considering the performance of each of its three parts as follows:

a) Promoting an efficient road transport system in the participating provinces is rated as Substantial.
A more efficient road transport system could be evidenced by reduced travelling time and lower travelling cost. While the ICR did not explicitly provide data on these variables, the project’s reported positive internal economic rates of return (ERRs), 21% for road upgrading and 60% for road reconstruction did reflect reduced vehicle operating cost and time savings, efficiency gains that were used to measure project benefits.
In addition, a socioeconomic study to evaluate the impacts of project road improvement activities on the project area households' income, access to health and education services and the poverty reduction included the following findings:

  • Visits to local towns and markets by household in the sampled project villages increased between 2004 and 2009 by 70%, against just a 40% increase by households in the sampled control villages (that did not benefit from the project).
  • Travelling times for local villagers to health facilities were not affected by the project as villagers used social
services provided within their own villages, or made shortcuts through the bush rather than walking along the
main road.
  • The ICR notes (ICR p.20) that travel time to secondary schools was reduced because the improved roads allow more convenient and faster motor vehicle rides, but does not provide specific evidence of this.


b) Promoting a safe road transport system in the participating provinces is rated as Modest.
At project completion, the ICR (p.18) reported more reckless driving along the improved roads, often under the influence of alcohol. The report did not provide concrete data on road accidents rates on project roads before and after the project, however. Additional information provided by the project team highlighted project interventions that could contribute to improved road safety. For example, before the project, two-way traffic was restricted dangerously to a single unmarked lane with the risk of head-on collisions on some sections. After the project, two separate lanes with edges protected by gabions were established. The project team also informed IEG that the project had improved driver visibility and drainage that had been poor before the project, leading to dangerous driving conditions.

c) Promote a reliable roads transport system in the participating provinces is rated as Substantial.
This IEG assessment considers that improved reliability can be measured by better road conditions that allow uninterrupted access to economic and social services. The ICR (data sheet) reported that overall road condition improved at project completion. The percentage of national roads reported to be in good or fair condition increased from 20% to 74% of the total during 2004-2009. It was also reported by the ICR (p. 18) that prior to the project, the residents of Kerema, the capital of Culf Province did not have all season road access to port Moresby, the nation’s capital. During the rainy season this road was impassable. With the project road improvements the villagers now have year round access to all the services of Port Moresby. The ICR reports (ICR pp. 18 and 41-42) some evidence from the project’s socioeconomic study of villagers enjoying higher incomes, although this cannot be attributed to the impact of the project alone:

i) 82% of the households living in villages close by the rehabilitated roads had monthly income increased as compared with an average of 73% of all the survey households with a increased monthly income;
ii) The percentage of households living below the poverty line (a monthly income of PGK500 or less) along the rehabilitated roads decreased from 62 percent in 2007 to 47 percent in 2009, even though the reduction could not be solely attributed to the project.

5. Efficiency:

At project preparation, economic analysis was carried out for all the national roads that would require periodic and specific maintenance (other than routine maintenance) or restoration as well as bridges that required replacement. As reported by the Project Appraisal Document (p. 18), Under the most likely traffic scenario, the Net Present Value (NPV) for the national roads was $36.5 million and average economic rate of return (ERR) was 30 percent, with the highest average ERR of 52% for road regravelling work and the lowest average ERR of 21% for road sealing work. Out of the 18 bridges initially identified, the project financed the 13 that were economically viable. At completion, their total NPV for these 13 viable bridges was $1.6 million. The sensitivity analyses also found that proposed project activities remained viable even with the standard cost increases or reductions in benefit used for such estimates.

At project completion, HDM4 model was applied for the economic analysis for the road works under the project, ERRs for all road works were higher than 12%. The analysis showed that the highest average ERR was 60% for roads reconstruction and the lowest average ERR was 21% for roads sealing works. The variations of ERRs were mainly due to the cost changes for road works. In these cases, the actual road reconstruction costs were lower than those estimated, hence, generating higher ERRs. Conversely, road upgrading works had higher actual costs resulting in lower, but still adequate rates of return. The efficiency of the project is rated as Substantial.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
30%
45.4%
ICR estimate:
%
78.9%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The relevance of the project objectives is substantial as is the relevance of project design. The efficacy of the project is assessed as substantial because the project objective of promoting an efficient, safe and reliable road system was achieved with only minor shortcomings, mostly in the area of road safety. With project efficiency substantial too, the overall outcome of this project is rated as Satisfactory.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:

A risk commonly faced by operations of this type is that the improvements in road conditions introduced by the project are not sustained beyond the project closing date due to insufficient political support, the lack of institutional arrangements and financial resources to ensure that necessary operations and maintenance continues. With regard to how this particular project might rise to challenges such as these, the ICR noted that:
i) There is a political will to maintain a good transport system in the country. The Government has recognized the importance of adequate transport infrastructure by calling for significant investments to rehabilitate and maintain the road transport infrastructure in its National Transport Strategy Plan;
ii) PNG now has a proper institutional set up to manage the country’s road network. A National Road Agency has been established to assume the responsibility of maintaining national roads throughout the country, although being relatively new and in needs of support for developing appropriate maintenance modalities;
iii) The ICR data sheet shows that the percentage of budget allocations at project completion for routine maintenance for the national roads was 67%, short of the target of 75%. However, additional information provided by the TTL indicated that the Government has indicated that funding for most road maintenance activities throughout the country are now coming from the Papua New Guinea Sustainable Development Program (PNGSDP), which takes royalties from the Ok Tedi copper and gold mine in Western Province. Yet, it is not totally clear how much funds could be provided to ensure that routine maintenance would be financed.

Given that there is proper institutional and some financial arrangement in place for road maintenance, but the key agency is new and untried and the funding for road maintenance is not firmly secured, the risk to development outcome is rated as Moderate.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:
For project preparation the Bank drew upon lessons from previous Bank-financed transport projects in the country. These included poor initial cost estimates leading to later overruns and unrealistic traffic volume projections that exaggerated the potential demand for a project’s services. To help avoid problems such as these, the project design not only included physical investments but also institutional strengthening and capacity building of both government roads agencies and private contractors. This helped ensure successful project implementation. The project quality was enhanced by a design that financed socioeconomic studies to record the poverty reduction impact of the project. As well as helping build political support for the need for road maintenance when the benefits of the project were recorded and shared with the stakeholders. These studies contributed significantly to understanding the results of the operation and the evaluation of them.

However, the evaluation took note the following shortcomings in the project preparation:
i) There was no activity directly addressing issues of road safety behavior and awareness, which lessened the project's capability to achieve the project development of objective of "promoting a safe roads transport system" in the participating provinces.
ii) The M&E framework was not comprehensive enough to track the achievement of project development objectives of "promoting an efficient, safe and reliable roads transport system". (for more details of IEG’s assessment of this project’s M&E see section 10).

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:
As reported by the ICR, the Bank team carried out regular supervision missions at least twice a year with additional special visits when needed. The Bank team worked closely with the Borrower to address all the implementation issues. During the whole course of the project, there were only three task team leaders, which provided some degree of continuity to the Borrower.

The project experienced considerable implementation delays mainly due to the lack of counterpart funding and the lengthy process of the Borrower's approval of consultant's contract for the project management. The lack of counterpart funding issue was not resolved until 2005/2006. While not the direct responsibility of the Bank, IEG concurs with the ICR (p. 21) that the Bank team could have been more proactive when it became apparent that counterpart funding was not forthcoming.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
As reported by the ICR (p. 22), Government performance improved over the course of project implementation, while the Government's commitment to the project development objectives remained strong. However, slow Government responses in some key areas contributed to the implementation delays. These included the slow processing of counterpart funding already mentioned, an issue only resolved in the fourth year of project implementation. There was also lack of inter-ministerial support to the Department of Works. The Ministry of Communication did not fulfill its responsibility of identifying local contractors who were to participate the training under the project. This resulted in the untimely transfer of the management of the project’s training component to the Department of Works. The National Executive Council’s protracted processing of approvals of large contracts also contributed to project delays. Consequently, project implementation was prolonged and the project closing date had to be extended.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
The project implementation agency was the Department of Works which was responsible for identifying roads and bridges to be improved under the project. The Department was also responsible for the implementation of the project activities. The project implementation experienced some initial delays due to the lengthy administrative process and the lack of counterpart funding at the earlier stage, however, the Department dedicated professional staff at all levels to manage project activities and took over management responsibility of the entire training component as a result of lack of inter-ministerial support to the project. In addition, the Department also contracted external consultant as project manager to support the implementation; All these efforts contributed to the satisfactory project implementation.

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The M&E framework included indicators at project output and intermediate outcome levels. Output indicators, for example, included the length of roads restored or maintained, and the number of bridges replaced. Intermediate outcome indicators included the number of provinces able to submit annual work programs and the number of provinces that actually did so, as well as the percentage of the total budget allocation going to routine road maintenance. The M&E framework also included impact indicators, such as the changing share of households in participating sample villages considered poor (with incomes below K100/month). However, there were no comprehensive outcome indicators to track the achievement of the project development objectives themselves. Thus, the M&E framework had no indicators on vehicle travelling time, travelling cost, road accidents rate. It therefore contributed little to the direct measurement of achieving the PDO. In addition, some of the targets, such as for the length of roads to be rehabilitated or maintained, were constantly revised during project implementation, undermining the credibility of the target initially established during the project preparation stage.

b. M&E Implementation:
As reported by the ICR, quarterly project management reports were prepared to monitor the progress on project implementation, however, the data collection was not always accurate and was at times problematic. For example, when routine maintenance on a road section was carried out more than once during the year, the length of roads being maintained were double or triple counted. This issue was addressed later on. The Department of Works also updated the project monitoring indicators annually, covering the physical, financial and procurement progress of the project. In addition, socioeconomic impact studies were launched in 2005 to quantify the project impacts.

a. M&E Utilization:
The M&E was utilized to monitor the project implementation progress. The Department of Works also utilized the information as a basis for reviewing its budget allocations for road maintenance and rehabilitation works, and for assigning contracts and signing off works.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
Environmental: The environmental category for this project was "B", the policy triggers were OP/BP4.01. The environmental category remained unchanged under the additional financing which was approved in 2007. As reported by the PAD (pp. 92-93), project preparation included a Sectoral Environmental Assessment together with an Environmental Management and Monitoring Plan that were prepared and were later revised in compliance with the Bank's safeguard policies. During project implementation, an Environmental Officer was appointed by the Department of Works to assist with monitoring environmental issues and to train and support the Department of Works provincial staff and contractors on environmental issues. There was no reported violation of Bank's policy in the ICR.

Social: OP/BP4.12 on involuntary resettlement was triggered and the "Environmental and Social Impacts Assessment Management and Monitoring Plan" was issued by the borrower in 2001. As reported by the ICR, during project implementation, a consultative process with all stakeholders was established to ensure common understanding of the possible impacts of the project and to foster local ownership of the project. The TTL provided additional information that there were a few disputes with land owners over very small pieces of land, but these were resolved in a manner that was in compliance with the project's safeguards documents.

b. Fiduciary Compliance:
Financial management: Current Bank policy on financial management is OP/BO10.02. To mitigate the financial management risks identified during project implementation, a full time international project financial controller was appointed by the borrower to implement and oversee internal controls and reporting. As reported by the ICR, the borrower complied with the financial management conditions specified in all financing agreements and satisfied Bank requirements. No ineligible expenditures were identified. However, from June 2005 to June 2009, there were delays in the submission of audit reports and later submission of withdrawals applications and Financial Management Reports to the Bank. All but one audit was unqualified; The one audit was qualified because of a specific issue related to the implementing agency's internal accounting and reporting of counterpart funding and expenditures, which did not affect IBRD/IDA funds/expenditures.

Procurement: As reported by the ICR, with the assistance of Employer's Project Manager (EPM ,service provided by a firm), the Department of Works carried out satisfactorily procurement for the project. However, the procurement process was very lengthy for those major contracts (above PNGK 10m) including EPM contract which required approval by the National Executive Council, resulting in significant delays of project implementation.

c. Unintended Impacts (positive or negative):
An unintended outcome reported by the ICR was the use of force account to complete the East-West Highway on Manus Island after the contractor for the work encountered problems and there was no interest from other private contractors for this piece of work due to challenging conditions. The execution of this contract by force account turned out to be successful. Based on this case, one other donor was planning to fund periodic maintenance using force account.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Satisfactory
Moderately Satisfactory
As assessed, there were some shortcomings with project design. The ICR also admitted that the Bank's team could have been more proactively dealing with some implementation issues. 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
This project was prepared and implemented in a very challenging environment., however, the project outcome was satisfactory. The ICR (pp. 23-25) drew the lessons on how to manage projects in fragile institutional settings low-capacity operating environments. The most important was related to risk management, as listed below:

i) Being rigorous and realistic in identifying potential risks at project design stage. For this project, a number of high risks , i.e, inadequate or late counterpart funding, potential political interference in drawing up annual work programs, inability of the government to acquire additional land, were identified at project preparation stage, therefore, enabled the project team to addressed these risks in a timely manner during project implementation.

ii) In countries where local agencies have only limited capacity for project implementation and little experience of working with the Bank, a project’s design should include specific actions to ensure that staff of these implementing agencies have adequate training on matters relating to Bank processes and procedures, and that the agencies themselves have regular access to qualified staff and consultants.

iii) Closely supervising the project through regular missions and site visits with government counterparts to manage the risk, including verifying implementation progress against agreed work plans, the quality of remote works, and compliance with safeguards. The close cooperation helped the Government understand better the Bank’s policy and timely resolve the issues jointly with the Bank team.

iv) Continually review and update the risk framework during implementation with attention to which risk mitigation measures are the most effective. Cultivate a shared approach that emphasizes that risk itself may not be inherently damaging, but unmanaged risk is.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

This ICR provided a clear story line on the project implementation including the additional financing. It also provided detailed information on the project outputs, intermediate outcome and social economic impact. In addition, the ICR also drew important lessons on how to manage the project in fragile institutional settings and low capacity environment. One of the shortcoming of the ICR was insufficient information was provided on the achievement of the project development objectives due the weakness of the M&E framework.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions