Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Albania Md Capacity Building & Support To Implement The Integrated Planning System


  
1. Project Data:   
ICR Review Date Posted:
10/19/2012   
Country:
Albania
PROJ ID:
P105143
Appraisal
Actual
Project Name:
Albania Md Capacity Building & Support To Implement The Integrated Planning System
Project Costs(US $M)
 7.19  5.52
L/C Number:
Loan/Credit (US $M)
 7.19  5.52
Sector Board:
Public Sector Governance
Cofinancing (US $M)
   
Cofinanciers:
Austria, Italy, Sweden, Switzerland, United Kingdom, EU, Netherlands
Board Approval Date
  01/17/2008
 
 
Closing Date
10/31/2009 09/30/2011
Sector(s):
General finance sector (40%), Central government administration (30%), General public administration sector (30%)
Theme(s):
Other public sector governance (20% - P) Public expenditure financial management and procurement (20% - P) Administrative and civil service reform (20% - P) Macroeconomic management (20% - P) Analysis of economic growth (20% - P)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Stefano Migliorisi
Clay Wescott Navin Girishankar IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The project development objective was to “ensure that the Government of Albania’s core policy and financial processes function in a coherent, efficient and integrated manner.” (Project Document, p. 6). The Grant Agreement adds that the PDO would be achieved “through the participation of the Recipient’s line ministries in the Integrated Planning System (IPS).”

    For this review, IEG uses this elaborated objectives statement. Also, for the purposes of evaluating efficacy (Section 4), IEG assesses two elements of the PDO. PDO1 relates to more integrated and coherent functioning of core policy and financial processes of the Government, and PDO2 relates to more efficient functioning of those processes.

    The project was restructured in August 2010, but the PDO was not revised.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

The original project had eight components:

    1. Stabilization and Association Agreement (SAA) Implementation and EU Assistance Programming and Reporting Capacities Strengthened – to support effective and timely implementation of the European Integration obligations in line with the commitments foreseen in the National Plan for the Implementation of the SAA (original cost US$ 0.82 million)
    2. Implementation of the Medium Term Budget Process (MTBP) – support to improving the MTBP process (original cost (US$1.41 million);
    3. Implementation of Public Investment Management – enhancing public investment management capacity (original cost US$ 0.97 million);
    4. Macroeconomic Forecasting – supporting the establishment of sound macroeconomic and fiscal forecasts as a basis for the multi-year budgeting process (original cost US$ 0.80 million);
    5. IPS Coordination, Oversight, Awareness, Communication – supporting the implementation of the Government IPS framework (original cost US$ 0.41 million);
    6. National Strategy for Development and Integration (NSDI) and Supporting Sector and Cross Cutting Strategies – improving linkages between national and sectoral strategic planning (original cost US$ 0.65 million);
    7. Aid Coordination – improving coordination of donor aid (original cost US$ 0.49 million);
    8. Monitoring and Reporting – establishment of IT systems (including Albanian Financial Management Information System or AFMIS) to support monitoring and reporting progress in strategy and MTBP implementation (original cost US$ 1.1 million).
In addition the components noted above, US$200,000 was set aside for contingencies and US$340,000 for the Trust Fund administration fee. Actual costs for these components were not provided in the ICR.

The components were revised during restructuring, although their number was kept constant. The second and third components were merged reflecting the Government’s decision to merge recurrent and capital budgeting processes and offices. The eighth component was focused more clearly on AFMIS. A new project management component was added to support capacity building of the Central Financing and Contracting Unit (CFCU), which functioned as the project implementation unit.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Cost: Financial project costs at appraisal were US$7.19 million. Actual costs were not included in the ICR.

Financing: The project was entirely financed through a Multi-Donor Trust Fund (amounts received by the Bank are indicated in brackets). Austria (Euro 500,000), Italy (Euro 368,787), the European Union (Euro 900,000), the Netherlands (EUR 1,000,000), Sweden (SEK 8,000,000), Switzerland (CHF 750,000), and the United Kingdom (GBP 1,500,000) contributed to the trust fund. These contributions included also the Bank’s administration fee which is not passed on to the Recipient.

The initial grant was equivalent to Euro 4.525 million, increased to Euro 5.525 in August 2010. An estimated Euro 1.45 million was undisbursed by project closing, mainly due to the lack of progress on AFMIS as discussed later in this review.

Borrower Contribution: The Government planned to contribute US$89,000. Information on the actual contribution is not provided.

Dates: The project was approved on January 17, 2008. A mid-term review was carried out in November 2009, and the project was restructured in August 2010 to adjust the results framework, slightly modify the components, and accept additional financing. The project’s closing date was extended twice from October 31, 2009 to September 30, 2010, and then to September 30, 2011 during the 2010 restructuring. The project closed as per revised schedule.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

IEG assesses the relevance of objectives is rated as substantial.

The PDO was -- and continues -- to be consistent with the Government’s reform agenda and the NSDI. The project was based on the IPS Implementation Plan produced by the Department of Strategy and Donor Coordination (DSDC) in April 2006. Furthermore, the objective is aligned with the Bank's current Country Partnership Strategy FY11-14, and specifically, contributes to the two pillars of the strategy -- (i) Accelerating Recovery of Economic Growth through Improved Competitiveness and (ii) Broadening and Sustaining Social Gains). These priorities are to be achieved by addressing fiscal stability issues and improving public service delivery, both of which are directly related to the PDO.

b. Relevance of Design:

IEG assesses relevance of design as modest based on its strengths and weaknesses:

  • The project’s design was overly complex and ambitious, given its short duration of three years and Albania's limited capacities (ICR, p. 6). It included eight components, 36 different activities, and numerous beneficiaries across all line ministries. In addition to the line ministries, project implementation relied on the Department of Public Administration (DOPA), the Training Institute for Public Administration (TIPA), the Ministry of Finance, the Department of Strategy and Donor Coordination (DSDC), and the Central Financing and Contracting Unit (CFCU).
  • The project introduced dual accountabilities -- ministries were responsible for agency-specific components and the DSDC for government-wide components -- that placed a premium on coordination. Administrative support provided by a Project Implementation Unit (PIU) at the Council of Ministers (staffed by only two civil servants) was not adequate to meet this challenge.
  • These above-mentioned factors -- complexity of design, limited capacity, and dual accountabilities -- contributed to inefficiencies in project management. As a result, the project implementation was substantially delayed. The pace improved only after the 2010 restructuring, which added the dedicated PIU.
  • There were some innovative aspects that sought to introduce flexibility during design and implementation. For instance, Component 2 (MTBP/Budget Capacity). It included subprojects to provide intensive, direct MTBP support to 4- 5 key ministries through TA over a 2-3 year period. The Ministries and activities were to be identified during implementation to enable the Government to adapt the project to changing circumstances. Flexibility is helpful for agency level change efforts, but not always appropriate for building highly technical systems across ministries and agencies. For instance, Component 3 (PIM) included support for software development for the AFMIS without clearly defined specifications upfront. However, there was not enough time to prepare these specifications and write the software code during the life of the project.
  • Overall, the intervention logic was simple: technical assistance and IT investments funded under the project (inputs) would allow the participation of Albania’s line ministries in the IPS (outputs), and this would ensure that the Government of Albania’s core policy and financial processes functioned in a coherent, efficient and integrated manner (outcome). However, the causal link between inputs, outputs and outcomes in the original design was not clearly established, and the results framework lacked a baseline on which to measure progress.


4. Achievement of Objectives (Efficacy) :



Efficacy is assessed against the goal of ensuring that the Government of Albania’s core policy and financial processes function in a coherent, efficient and integrated manner through the participation of the Recipient’s line ministries in the Integrated Planning System (IPS).

Outputs: All outputs, with the exception of the development of AFMIS and HRMIS, were achieved before project closing. These include the completion of training (i.e., on budget planning, analysis and monitoring; on external assistance); training needs assessments and strategy; redesign of documents, manuals or strategies (on MTBP, policy impact assessments, and debt management); and regulations (i.e., role of DSDC in coordinating external aid). Nevertheless, the AFMIS was not completed as direct (sole source) contract negotiations with the developer of the Treasury IT system failed in April 2011, due to the refusal by the contractor to accept standard World Bank terms.

These outputs generally led to some improvements in policy and financial processes at the government-wide and line ministries, although further strengthening of coordination across ministries is needed:

  • The 2010 SIGMA assessment of public sector performance in Albania recognized progress in the government's strategic planning, while at the same time, pointing out the weakness in central coordination of different line ministries' legislative initiatives. This finding is corroborated by a PEFA Assessment which also acknowledges improved policy-based budgeting and coordination of the budget planning process.
  • Also, medium term budgeting process improved as a result of better integration of strategic planning, MTBP, and budget allocations. The PEFA Indicator PI-12 “Multi-year perspective in fiscal planning, expenditure policy and budgeting” has show marginal improvements from C in 2006 to C+ in 2011. Yet, most sector strategies remain poorly linked to the MTBP and lack costing and clear performance frameworks.
  • Management of external assistance also improved, but not as much as claimed in the ICR. A 2011 Paris Declaration Survey found that slow progress has been made since 2007 regarding alignment and harmonization, with a decline (rather than an increase, as claimed in the ICR) in the use of country systems.

Outcomes (against PDO1 and PDO2):

Policy and financial processes are more coherent and integrated (PDO1): The restructured results matrix defined the outcome in a coherent and integrated manner as a clear link between each Ministry’s plan defining policy goals and objectives and the NSDI. The ICR found that, while all Ministries produce sector strategies, these often lack clear links to the NSDI and are not costed. The efficacy of this PDO was modest.

Policy and financial processes are more efficient (PDO2): The improved efficiency of the GoA policy and financial processes was instead defined as a strengthened respect of the MTBP ceiling by each Ministry with a deviation under 6 percent. In reality the average deviation was reduced substantially, from 21.6% in 2008 to 5.4% in 2011. The efficacy of this PDO was therefore substantial.


5. Efficiency:


The ICR's discussion of efficiency was found to be wanting. The project appraisal did not include a formal cost benefit analysis, even though it should have been possible to provide information on unit rate norms for technical assistance. IEG is not able to calculate and assess these rates in the absence of detailed data on days of technical assistance and their cost.

It is also possible to analyze other aspects of design and implementation that affected the project’s efficiency:

  • As noted earlier, the design was ambitious -- covering a number of components and activities in a weak capacity environment. This design contributed to the limited efficiency of project management and monitoring and evaluation, and to a rather slow start.
  • The number of Ministries and Agencies involved in the project placed a premium on coordination. The PCU was unable to deliver until the restructuring.
  • On the positive side, the project combined the support from several donors in a single MDTF, thus achieving efficiency gains compared to the alternative of separate grants.

On balance, IEG assesses the project’s efficiency as modest.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Through this operation, Albania's public sector has strengthened strategic planning and medium-term budgeting processes. As a result, the country has made substantial progress towards achieving the goal of making core policy and financial processes of the Government more efficient, yet more modest improvements in making these processes more integrated and coherent (for instance, development of sector strategies). The objectives of the project were assessed to be substantially relevant, even though relevance of design and project efficiency were modest. Following IEG-OPCS's Harmonized Evaluation Criteria (and IEG's criteria for evaluating split ratings), the ICRR assesses project outcome to be moderately satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

IEG concurs with the ICR’s assessment of risk to development outcome as moderate. The IPS reform program has been under implementation for a long period of time and the GoA has confirmed its commitment to it through a continuous effort. The importance of IPS for the SAA with the EU is an additional guarantee that efforts will continue. As stated in the ICR, a second IPS project has been requested by Government and is currently under preparation, thus guaranteeing continuation in donor funding of the program. The combination of these factors minimizes the risk the overall risk of a possible policy reversal. However, the resistance to computerization and insistence on sole source contracts both show cause for concern.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:

The project's clearly defined objectives sought to address key PFM issues in the country, and it was able to provide support from multiple donors from a single fund with a common set of procedures. Yet, the original results framework did not provide credible causal links among inputs, outputs, and objectives, and was not operationally useful. The original components and results framework were not “focused on achieving project objectives.” (ICR, p. 26). The overly complex design, in a weak capacity environment, could have been simplified and better fitted to the Albanian context. By the same token, high risk components such as the AFMIS were added late in the preparation process and were found to be under-designed. With a more rigorous appraisal of the operation, the Bank team could have helped avert some of the implementation challenges evident over the project's life.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

The Bank supervised the project on a regular basis (3-4 times a year) and provided continuous support from the Albania office. It was able to identify the design problems that caused the initial implementation delays, albeit quite late in the project cycle (only a little over one month before the original closing date). Nevertheless, the Bank's proactive support contributed to the project achieving a good part of its original objectives, during the one year extension granted as part of the restructuring. By the same token, the Bank should have taken a tougher stand on the sole sourcing of the AFMIS contract. The lack of ISRs till 2010 and the constant shifting of PDOs being monitored are two additional weaknesses of Bank supervision.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government of Albania demonstrated substantial ownership and commitment to the project, as shown by its continued effort over the years, achieving substantial results in the efficiency of its financial processes. The implementation issues that caused the initial delays were addressed through restructuring and the GoA has requested support from the Bank for a follow on operation. The long time lags in project preparation and restructuring were mainly due to the need to reach agreements with a large number of donors (seven in this case). There were no fiduciary issues mentioned in the ICR or the other documents examined by IEG. Stakeholders were consulted during implementation through an M&E workshop in March 2011.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:

The performance of the Ministry of Finance was rather modest, which was at least partly due to design issues. The outputs missed by the project (i.e., Human Resource Management Information System - HRMIS and Albanian Financial Management Information System - AFMIS) were due to lack of focus by the implementing agencies concerned and to the low capacity of the Ministry of Finance in handling large IT projects. In addition, the problems with AFMIS were partly caused by the implementing agency’s decision to proceed with a sole source contract with a supplier that eventually refused to accept standard conditions for contracts in Bank funded operations. Performance improved after restructuring when the CFCU became the project's implementing agency.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The quality of M&E design was rather poor. As noted in the ICR (p. 6), the original results framework included 3 goal indicators, 9 purpose indicators, and 38 output indicators, and linkages between output and higher level indicators were weak. The output indicators had no baseline values. This flaw was addressed during the 2010 restructuring that introduced a new results matrix with 2 outcome and 19 output indicators. The revised M&E matrix was an improvement, but still included a relatively large set of indicators.

b. M&E Implementation:

The three ISRs show a thorough monitoring of the results matrix, as the original one had no operational value and was not used. The goal indicators used in the ISRs vary without any explanation.

a. M&E Utilization:

There is no information in the ICR about the utilization of the M&E system. Donors contributing to the MDTF made use of the reports produced after restructuring, after they insisted for the replacement of the original framework that was not allowing them to properly understand project performance.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

There were no safeguard issues highlighted in the available project documents.

b. Fiduciary Compliance:

There were no fiduciary compliance issues identified in the project documents. The project did not use Albania’s country systems because donors asked that Bank processes be used.

c. Unintended Impacts (positive or negative):

The project indirectly supported the launch of the new computerized Treasury System that had been previously installed but not used, through a joint letter to the Minister of Finance from all donors supporting it.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Unsatisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The main lessons that the ICR and IEG identified are as follows:
  • Complex design in a low capacity environment should be avoided at all costs. While it is tempting to combine many components into a single operation for possible economies of scale in preparation and supervision, the reality is that the pace of implementation can suffer, and the savings in preparation are often translated into more intensive supervision and a subsequent re-design of the project.
  • Projects covering a short time span should avoid complex but under-designed components. Major IT investments such as the AFMIS are more likely to succeed if planned in sufficient detail before the implementation.
  • Lack of competition in contract award can have a negative impact on efficiency or efficacy, and should be avoided for large contracts or important project components.
  • An excessive number of donors in a multi-donor trust fund established to support a specific project can add unnecessary complexity to the project itself.
  • When PIUs are staffed by civil servants, there should be sufficient numbers of staff with the required skills, and sufficient time freed up from other responsibilities to be able to effectively carry out project management functions.
  • The Bank’s design, supervision, monitoring and reporting on trust funded projects needs to improve.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR offers a candid review of project performance, and provides some useful information. However, it did not provide any information on the project’s efficiency, and included a few erroneous indicators (e.g., on Aid Effectiveness). It also did not include details on the project’s restructuring in the datasheet, which also contains some discrepancies (the date indicated in the datasheet is November 2010 but August 2010 in the main text). Finally, the ICR does not provide any feedback from the donors funding the MDTF, or any information on planned and actual costs by component (Table in p. 18 is not filled out).

Considering these limitations, IEG assesses the quality of the ICR to be unsatisfactory,

a. Quality of ICR Rating: Unsatisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions