Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Public Sector Financial Management Reform Support


  
1. Project Data:   
ICR Review Date Posted:
04/15/2013   
Country:
Georgia
PROJ ID:
P063081
Appraisal
Actual
Project Name:
Public Sector Financial Management Reform Support
Project Costs(US $M)
 15.0  10.76
L/C Number:
CH205
Loan/Credit (US $M)
 3.0  1.38
Sector Board:
Public Sector Governance
Cofinancing (US $M)
 9.1  8.74
Cofinanciers:
DFID, Netherlands, SIDA
Board Approval Date
  02/16/2006
 
 
Closing Date
03/01/2010 03/01/2012
Sector(s):
Central government administration (100%)
Theme(s):
Public expenditure financial management and procurement (29% - P) Administrative and civil service reform (29% - P) Other accountability/anti-corruption (28% - P) Other public sector governance (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Clay Wescott
Rene I. Vandendries Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:


    "The objectives of the Project are to support the Recipient's Program: (a) to strengthen the institutional capacity of key ministries and agencies to use more effectively and efficiently public resources; and (b) to improve accountability in the use of public resources." (Financing Agreement, p. 5).

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

Component costs at appraisal are taken from the Project Appraisal Document, Annex 5. There were four components:

1. Medium Term Expenditure Framework (MTEF) and budget management (cost at appraisal: US$2.27 million, actual cost: US$3 million)

Supported establishing the MTEF as the principal decision-making tool for linking budget resource allocations to policy objectives and program priorities, and for ensuring greater budget predictability for line ministries; and strengthening budget planning and management processes within the Ministry of Finance (MoF) and line ministries so that the budget is implemented consistent with the strategies and priorities identified in the MTEF.

2. Treasury reform and budget execution (cost at appraisal: US$7.83 million, actual cost: US$5.22 million)

Supported the Treasury in leading reforms in accounting standards and regulation, cash management, procurement, introducing a professionalized public accounting community, and developing a modern public financial management information system (PFMIS).

3. Development of the human resources management information system(HRMIS) (cost at appraisal: US$2.11 million, actual cost: US$0.123 million)

Supported a mechanism to collect, process and update data on the size and composition of the civil service, an automated payroll for the civil service to ensure a proper link between wage expenditures, the budget and the MTEF, and development of a feasible, medium-term civil service reform strategy

4. Improvement of public audit and oversight of public financial management (cost at appraisal: US$1.92 million, actual cost: US$1.79 million)

Supported the transformation of the Chamber of Control into a supreme external audit institution, with human resources capacity responsive to its needs, an audit methodology in line with international best practice, and greater transparency leading to more informed public discussion of its activities and findings.

In addition to the four main components, US$0.67 million was allocated to support project management.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

The closing date was extended on 02/25/2010 for two years to 03/01/2012 following the mid-term review to accommodate delays in implementing the PFMIS. Subsequently, the borrower decided to finance the PFMIS with its own resources. Funds were redirected to other needs such as equipment for the Finance Academy and Procurement Agency. With a large remaining balance in the Single Treasury account, cancellations were made by the Netherlands Embassy (US$730 thousand of US$1 million), the Swedish International Development Agency (SIDA - US$1.3 million of US$4.5 million), and the International Development Association (SDR 1.2 million or about US$1.62 million of US$3 million).

Although the PDO was not revised, PDO indicators were revised November 2006. There were two large reductions in component costs. The cost for the component supporting treasury reform and budget execution was reduced because of the decision by the borrower to finance the PFMIS with its own resources; the cost for the HRMIS component was reduced because of the reduction in scope. The cost for the MTEF and budget management component increased because of a reallocation of funds to support capacity building of the Finance Academy, which was established in 2010 to support information and communications systems used by the Ministry of Finance, and to support selected investments in information and communications technology infrastructure. In addition to the cost of the four components, US$0.67 million was allocated to support project management, and the actual cost was US$0.63 million. The borrowers contribution estimated at appraisal at US$900,000 was actually US$640,000. The Government had committed to cover six percent of project costs, so when the cost was reduced, the Government contribution was proportionately reduced.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Substantial. The objectives were relevant at the time of appraisal, aligned with the FY06-09 Country Partnership Strategy (CPS) objective of improving efficiency in public services, and with the Government's Strategic Vision for Public Financial Management Reform. The objectives remain relevant to the FY10-13 CPS outcome of improved public resource management.

b. Relevance of Design:

Substantial. There is a clear statement of objectives. Intermediate and final outcomes are clearly stated in the PAD, and further refined in 2006 as discussed above. The causal chain between Bank funding, intended outputs and outcomes is convincing, taking into account complementary resources from the Poverty Reduction Support Operation series, and parallel support from DfID, Netherlands, European Union, GTZ, IMF, UNDP, USAID, and the US Treasury that supported the same and related objectives. For example, support from the Bank and other donors in macroeconomic and fiscal forecasting, budget and public investment management, and inter-governmental finance would plausibly lead to the expected improvement in a multi-year perspective in fiscal planning, expenditure policy and budgeting. The work built on successful reforms in, inter alia, strengthening tax collection, customs, and public and civil registries (World Bank, 2012. Fighting Corruption in Public Services: Chronicling Georgia's Reforms). Complementary reforms supported by other operations also included the Treasury Single Account, a Government Financial Statistics (GFS) 2001 compliant classification introduced for the 2008 budget, and improvements on procurement, including the launching of the State Procurement Agency website to provide greater transparency, and the adoption of the new Law on Public Procurement.


4. Achievement of Objectives (Efficacy) :


To strengthen the institutional capacity of key ministries and agencies to use more effectively and efficiently public resources. Substantial. According to the ICR, most indicator targets have been achieved. There have been improvements through a multi-year perspective in fiscal planning, expenditure policy and budgeting, expected to be confirmed by the next PEFA assessment of PI (PEFA indicator) 12 . Likewise, improvements are expected to be confirmed by the next PEFA assessment in predictability of the availability of funds for commitment of expenditures (PI-16), In addition, continuation of a top "A" rating for orderliness and participation in the annual budget process is expected to be achieved (PI-11). However, the target of achieving a functioning PFMIS has not been achieved. The MOF cancelled an international competitive bidding (ICB) tender in 2010, and proceeded to implement an in-house solution. This solution does not fully implement the recommendations of an independent assessment that preceded the decision to take this alternative approach. While about half the modules of an in-house system have been implemented, there are doubts as to whether the expected quality and level of integration can be achieved with the current approach. Looking at Georgia's overall Country Program and Institutional Assessment score for the quality of budget and financial management, the rating was unchanged from 2006-11 (4 out of 6).

To improve accountability in the use of public resources. Substantial. Again, most indicator targets have been achieved. Improvements are expected to be confirmed by the next PEFA assessment in scope, nature and follow up of external audit (PI-26), and public access to key fiscal information (PI-10). The target was achieved of having a supreme audit institution in place and functional, with supporting legal framework. Although the original design target was to establish an HRMIS for the civil service, this had to be revised to cover only 5,000 staff in the Ministry of Finance system following a reorganization of the Public Service Bureau in 2006. In addition, the MOF implemented a payroll system covering the entire civil service of about 80,000. The revised target was achieved. The target of increasing staff capacity for audits could not be measured. No improvement is expected in quality and timeliness of annual financial statements (PI-25), Looking at Georgia's overall Country Program and Institutional Assessment score, the rating for quality of public administration was unchanged from 2006-11 (3.5), and the rating was unchanged (3.5) for transparency, accountability and corruption in the public sector over the same period. However, Georgia has increased its score on the Open Budget Index from 34 to 55 (out of 100) from 2006-2012, with notable progress in publishing its pre-budget statement, executive budget proposal, enacted budget, and audit report.

5. Efficiency:

Modest. The PAD does not provide a cost benefit analysis summary. However, it states that the return on project investment would come from better performing public institutions. Specifically, streamlined budget preparation and implementation would lead to greater efficiency in budget transactions. The PFMIS would enable better control of financial and human resources. Better cash management would reduce idle cash balances, and better budget execution would reduce delays in payments and in turn the cost of goods and services to the government.

    The ICR doesn't measure any of these possible investment returns, nor does it discuss efficiency. However, there were features of the design and implementation that may have reduced efficiency. For example, the Mid-Term Review found a number of implementation shortcomings, including a perceived lack of commitment and shift in focus in the vision for PFM reform, delay in procurement and PFMIS development, turnover and lack of capacity in the Project Implementation Unit, and unsatisfactory project monitoring and reporting. Design weaknesses such as overly complex design, and unclear lines of accountability, may have also weakened efficiency.

    a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
%
%
ICR estimate:
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The objectives were relevant to Bank and Government strategies both at appraisal and at closing. The design was also substantially relevant, with a convincing causal chain linking funding from this, other Bank operations, and those of other donors, and the intended outcomes. Efficacy of objectives was also substantial, with most indicator targets achieved or partially achieved. Efficiency was modest, with no evidence provided to measure savings achieved due to greater process efficiency, and a number of implementation shortcomings that may have reduced efficiency in project delivery.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The ICR points out there is a risk that the in-house development of the PFMIS may not achieve the desired level of quality and system integration, leading to weaker financial and personnel controls, less effective cash management, and no reduction in payment delays. There is also the risk that capacity built up with support from the operation will be lost due to personnel changes. This risk may be mitigated through ongoing training using the Finance Academy and the State Audit Office Training Center, and twinning with the Swedish National Audit Office.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:

Lessons learned from previous and ongoing operations from the Bank and others were incorporated. Significant co-financing was mobilized, helping to leverage the small IDA credit with co-financing using trust funds covering 60% of expected costs at appraisal. The pooling arrangement for donor funding was new to Georgia, as was the use of country systems for disbursing pooled funds in line with Government policy. It was appropriate for a project intended to strengthen Government financial management systems.
Despite these positive features, there were shortcomings. A key feature of the design, the installation of an international procured PFMIS, was reportedly only supported by a resident IMF advisor and a small group of technical staff in Treasury, and not more widely by senior management in MoF. In addition, the complexity of the project design with many activities carried out in parallel by different agencies led to capacity challenges for the Government. This, along with unclear lines of accountability at lower levels in ministries may have weakened implementation. These shortcomings were highlighted in both Project Concept Note and Quality at Entry Review meetings, but not addressed in the final design, and contributed to unsatisfactory implementation progress in 2009 and 2011.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

Supervision missions and the Mid-Term Review (MTR) raised implementation issues with authorities, but could have downgraded the IP rating much earlier to indicate that performance was well below expectation. More extensive consultations in the early years of implementation might have uncovered earlier the thin support in MoF for the PFMIS, and addressed it at an earlier stage before carrying out a time consuming and ultimately fruitless ICB. The external assessment of counterpart information technology capacity was a useful suggestion from the Bank, but it came too late in the project cycle for all the recommendations to be implemented. The continuity of a Task Team Leader (TTL), trusted by senior government and political leaders, and based in Georgia played an important role in the building borrower capacity, and in facilitating the extensive consultative process with key stakeholders.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government's Strategic Vision for Public Financial Management Reform adopted in 2005 and updated in 2009 was a credible, overarching framework to guide reforms, including those not supported by this operation such as the Treasury Single Account, Government Financial Statistics (GFS) 2001 compliant classification, and the new Law on Public Procurement. However, the unexpected burdens of the military conflict with Russia and the regional financial crisis reduced the priority on financial management reform, and may have contributed to the delays on project implementation.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

The three implementing agencies achieved most targeted outputs despite the difficult context of war and financial crisis. However, there were shortcomings that contributed to delays, and to partial or non-achievement in some areas. The MoF lacked prior experience in implementing Bank funded projects. The Government's initial desire to purchase an off-the-shelf PFMIS solution was based on its limited capacity, and limited knowledge of the possibilities. By 2010, Government information technology staff had proven their ability to develop in house system modules judged to be better value and more appropriate to the context; thus the Government's decision to cancel the ICB was reasonable. Likewise, the initial plan for a Government-wide HRMIS was based on the limited experience of senior staff that had just taken power, and their preference for a centralized personnel management system. As capacity increased, the decision was made that the role of the Public Service Bureau should be to set the policy framework, while leaving personnel management to the ministries. This led in turn to the dropping of the HRMIS, and substituting for a less ambitious system covering only finance staff. And multiple changes in leadership of the Chamber of Control affected implementation of work under its domain.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

PDO and intermediate outcome indicators were clearly stated with baselines and measurable target values in the second Implementation Status and Results Report (ISR) of October 2006, and linked to the relevant PEFA indicators that became available at that time. A minor shortcoming is that one PDO measures PI-12 (iii) and another measures the entire PI-12, which is double counting.

b. M&E Implementation:

In the fourth ISR of June 2007, the human resources management progress reporting was changed in scope from the original focus on an HRMIS covering the entire civil service, to a system covering the MoF system of 5,000 staff (out of a total of about 80,000 in the civil service). The reason for this is that the scope of the project was reduced after a reorganization of the Public Service Bureau in 2006 which made it impossible to implement the more ambitious original scope in this area. The ICR states that there were delays in reporting leading to monitoring weaknesses in the period 2007-9. There was an improvement after that, but it was never satisfactory.

a. M&E Utilization:

Because of the sound design, the M&E framework, allowed for comprehensive reporting of results in the ICR. However, the reporting delays discussed above led to the under-utilization of M&E findings during project implementation.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

There were no safeguard issues under this Category "C" operation.

b. Fiduciary Compliance:

Financial management was acceptable to the Bank. Government counterpart funding reached the agreed six percent of project cost by project closing, with some delays in meeting commitments during implementation. Planning and budgeting arrangements were acceptable to the Bank, with some delays. There were no misprocurements, but there were some issues: not all bids were technically sound, some requirements were prepared with specific vendors in mind, and bids evaluation was not fully documented.

c. Unintended Impacts (positive or negative):

While the pooled funding arrangement was intended to provide flexibility to the borrower, there were unintended shortcomings, including limited flexibility to donors in adjusting disbursements to changing conditions, and challenges in meeting donor reporting requirements.

d. Other:
none



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

Key lessons in the ICR are supported, inter alia as follows:

1. Complex reforms should be approached by creating a decision-making environment that encourages experimentation and feedback loops in learning, where there is continual results monitoring and redesign based on experience. Such redesign may be needed due to changes in Government priorities, as well as actual results which may be unexpected. Project designs should be adaptable enough to allow for such changes, while ensuring that the revised targets are as significant and meaningful as the original ones.

2. Design complexity should be managed to be within the capacity of implementing organizations. When many reforms are needed, it is worth considering to carry them out sequentially to avoid overly complex arrangements.

3. Major interventions should be preceded by consultations where key stakeholders come up with a prioritized set of locally defined problems, and recommended solutions. Politically acceptable solutions come from a gradual process of step-by-step experimentation, and engagement with many stakeholders playing many different roles. This engagement comes from long conversations with people, both individually and in workshops. PFMIS development and other key initiatives should move ahead as part of such processes, so that key stakeholders holding different roles agree that the initiatives are the proper response to identified problems. While Georgia was not categorized as a fragile and conflict affected state (FCS), the period after the Rose Revolution was characterized by the same diversity of views, capacity gaps, and uncertainty often found in FCS contexts. The deep, consultative approach used in Georgia may be particularly relevant in such situations.

One other, crucial lesson is important. To ensure effective coordination of support to complex, institutional reforms, it is very useful to have a TTL trusted by senior government and political leaders, leading the consultative process from the Bank side. There are also important benefits if such a TTL stays with the project from design to completion.


14. Assessment Recommended?

Yes
Why?


During the period after a new government took power in 2004, there were many different ideas among key stakeholders on new directions to follow. It would be important to better understand how the Bank evaluated these different possibilities, and helped build a coalition within both Government and key development partners on a course of action.


15. Comments on Quality of ICR:

The ICR provided a frank analysis of achievements and challenges. It could be improved by adding a discussion of the efficiency in use of project resources.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions