Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Integrated Financial Management Iii - Technical Assistance Project


  
1. Project Data:   
ICR Review Date Posted:
06/20/2012   
Country:
Guatemala
PROJ ID:
P066175
Appraisal
Actual
Project Name:
Integrated Financial Management Iii - Technical Assistance Project
Project Costs(US $M)
 33.20  46.91
L/C Number:
L7104, L7522
Loan/Credit (US $M)
 29.75  44.53
Sector Board:
Public Sector Governance
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  03/14/2002
 
 
Closing Date
09/30/2006 06/30/2011
Sector(s):
Central government administration (70%), Sub-national government administration (30%)
Theme(s):
Tax policy and administration (33% - P) Public expenditure financial management and procurement (33% - P) Other accountability/anti-corruption (17% - S) Decentralization (17% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Clay Wescott
Jorge Garcia-Garcia Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The project sought "…to extend and deepen the reforms being pursued under the ongoing Integrated Financial Management II Project, Loan GU-4269 (IFML II), with the aim of increasing the effectiveness, efficiency, and transparency of public sector financial management and control." (Project Appraisal Document (PAD), p. 2 and Loan Agreement, p. 15)

    The new loan agreement of March 6, 2009 defines the objective of the Project as “to extend and deepen the Borrower’s financial sector reforms being pursued under the Original Project with the aim of increasing the effectiveness, efficiency, and transparency of public financial management.” (p. 5)

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 03/27/2008

c. Components:

1. [Part A in loan agreements of February 10, 2003 and March 6, 2009] Consolidating and institutionalizing Sistema Integrado de Administracion Financiera (SIAF, Integrated Financial Management System) within the Ministry of Finance (US$9.18 million at appraisal, US$14.6 million after additional financing, US$ 17.78 million at completion) Supports completing SIAF installation, deconcentrating it to the budget execution level, completing the SIAF sub-systems, and deepening budget reforms. The additional financing supports multi-annual results-oriented budgeting, gradual incorporation of international standards for accounting, budget and public debt, capacity strengthening in Ministry of Finance (MINFIN) to take over management and operation of SIAF, expansion of e-procurement (GUATECOMPRAS) system, and transfer of SIAF and GUATECOMPRAS to MINFIN operation.

2. [Part B in loan agreements of February 10, 2003 and March 6, 2009] Strengthening municipal governments' financial management (US$ 12.13 million at appraisal, US$18.03 million after additional financing,US$14.32 million at completion) Supports adaptation of SIAF to municipalities (aimed at all 330 with exception of the capital, which already had its own system), and their compliance with national laws. The additional financing supports development and implementation of the updated financial information system, and capacity strengthening in MINFIN to support and advise municipalities.

3. [Part C in loan agreement of February 10, 2003 and does not appear in LA of March 6, 2009] Modernizing Human Resources Management Information (US$1.33 million at appraisal, US$1.39 million at completion) Supports a system of payroll liquidation, a personnel records system, and improved human resources capacity.

4. [Part D in loan agreement of February 10, 2003 and Part C in loan agreement of March 6, 2009] Strengthening of the Comptroller General's Office (CGC - US$3.35 million at appraisal, US$8.75 million after additional financing, US$6.45 million at completion) Supports modernizing CGC, creating internal audit units in all ministries, most decentralized agencies, and 50 municipalities, enhanced transparency, and training. The additional financing supports strengthening municipal investment planning, integration of the national public investment system (SNIP) with SIAF, and strengthening non-financial audit capabilities, and CGC internal functions to improve performance and promote public participation in oversight, including creation of a complaint center.

5. [Part E in loan agreement of February 10, 2003 and does not appear in LA of March 6, 2009] E-Government (US$1.50 million at appraisal, US$1.90 million at completion) Supports strategic planning, legislation, a diagnostic on telecommunications infrastructure, and strengthening the Government's Internet portal.

6. [Part F in the loan agreements of February 10, 2003 and March 6, 2009] Project coordination (US$3.07 million at appraisal, US$4.47 million after additional financing,US$2.33 million at completion) Supports improved operational capacity of the Ministry of Finance.

Two new components were added under the additional financing plan:

7. [Part D in the loan agreement of March 6, 2009] Strengthening and integrating planning and public investment procedures and systems (US$1.37 million after additional financing, no expenditure at completion) Supports strengthening the Secretariat of Planning and Programming (SEGEPLAN, updating Sistema Nacional de Inversion Publica (SNIP), developing an e-learning system in sub-national public investment, and strengthening the Public Investment Directorate to manage SNIP and to integrate SNIP and SIAF.

8. [Part E in the loan agreement of March 6, 2009] Designing and institutionalizing a permanent knowledge management system in the area of public expenditure management (US$0.50 million after additional financing, no expenditure at completion) Supports work relating to planning, monitoring and evaluation, financial management, public procurement, human resource management, and auditing, including requirements under the SIAF model.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

The total estimated cost of the project at appraisal was $50.55 million. The total Bank financing was $49.75 million. The project was approved March 14, 2002, and became effective April 18, 2003 for an initial amount of $29.75 million. Additional financing of US$20 million was approved in March 27, 2008, and the loan became effective March 19, 2009. The long gaps in both cases between approval and effectiveness were due to the traditionally slow process of congressional approval. A restructuring of the additional financing was approved by the Bank in March, 2011. Actual loan disbursement was 89 percent of the appraised amount including additional financing. The actual government contribution of $2.38 million by project closing was 69 percent of the appraised amount.

At the component level, there are differences between estimated and actual costs for various reasons. In the case of Component 1 (SIAF), the reason for the larger actual cost is the decision to build a centralized, cloud server. This in turn reduced cost for Component 2 (municipal finance), since municipalities could access the centralized server via Internet, and didn’t need to finance their own servers. Component 4 (CGC) cost less than planned because of delays at the beginning due to low absorption capacity and weak leadership; as these challenges were addressed, progress was made, but the initial delay meant that the total budget wasn’t spent. Component 6 (Project Coordination) cost less because of the new thrust following the additional financing commitment to turn over project work to mainstream government departments. Thus, consultants that had been charged to this component were charged to other components. No project funds were used for Components 7 and 8. In the former case, the counterpart went ahead without Bank funds because of the delays in obtaining those resources, linked to the slow effectiveness of the additional financing due to the delays in obtaining approval in Congress. Regarding Components 7 and 8, activities were covered with EU funds (grants). A project preparation Japan Policy and Human Resources Development (PHRD) grant was made available in 2007 (TF057798) to support preparation of a separate project on Public Sector Modernization and Governance included in the CAS for that period. Due to Government constraints in the fiscal side, the Government and World Bank decided to discontinue the preparation of such project and replaced it with the additional financing for the SIAF III project. Part of the PHRD funds were used to support some related consultancies but most of the grant money was canceled.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

The objective is substantially relevant to addressing challenges of inadequate human resources and personnel management, weak tax administration and low tax rates by regional standards, and an overly centralized, rigid and non-transparent public financial management system biased toward spending in the capital region rather than in the poorer rural areas. The objective builds on major reforms of public financial management that had been supported by four successive Administrations by the time of appraisal, including actions taken to provide access to government accounts through the Internet and kiosks. It is relevant to the Country Partnership Strategy for FY09-12 and its objective of enhancing fundamentals, which includes enhancing fiscal space for priority public spending, strengthening integrated public financial management and tax administration, enhancing transparency and efficiency of public spending, and improving decentralized monitoring and evaluation and results based budgeting. The global economic crisis, and domestic political factors which have reduced progress in some areas, have added heightened priority to reforms supported by this operation in improved management and transparency of public spending.

b. Relevance of Design:

The design is substantially relevant. There is a clear statement of objectives, linked to intermediate and final outcomes, with a convincing causal chain linking funding from the Bank with intended outcomes. The design was built on achievements and lessons learned from two previous operations, IFM1 and IFM2, related operations such as Tax Administration, Judicial Administration Reform, and Private Participation in Infrastructure, and recent AAA including the Country Procurement Assessment Report (1999) and the Draft Country Financial Accountability Assessment (2002). The additional financing and restructuring built on additional AAA, e.g. Public Expenditure and Financial Accountability assessment (PEFA - carried out 2009, published 2010).


4. Achievement of Objectives (Efficacy) :


The achievement of efficacy of objectives is measured by considering achievements in improving effectiveness, efficiency, and transparency of public sector financial management and control.

1. Effectiveness. All PDOs were achieved, including transfer of SIAF functions to MINFIN, use of SIAF to adjust budget proposals, use of SIAF by 100% of municipalities, expansion of internal audit, SNIP integrated with SIAF, and related training and real time reporting of financial information on government websites. Most intermediate results indicators were also achieved linked with SIAF, municipalities, Office of Comptroller General (CGC), planning and public investment systems, and knowledge management. Even small components for human resources and e-government achieved progress, although the latter wasn't critical to achieving the PDO. After project completion, many project consultants, particularly ones carrying out IT functions, were kept on by the Government under an enhanced compensation package. Other consultants were shifted to related projects of other donors, on the understanding that they will shift to government employ in the medium term.

Despite the achievements in rolling out SIAF systems to municipalities, the 2009 Public Expenditure and Financial Accountability (PEFA) assessment reports that less than 1/4 of municipalities complied with submission of information on budget management to MINFIN, and municipal fiscal information isn't consolidated. The Task Team Leader (TTL) reports that at present, 90% of municipal expenditures are covered by the SIAF systems, and thus accessible to MINFIN. There have also been modifications to the Municipal Code in May 2010 with provisions to better align federal and municipal budget preparation cycles. The 2009 PEFA finds that there have been "very important advances in almost all the aspects of public financial management"; for example, there are minimal differences between aggregate expenditure and the approved budget, and public access to key fiscal information is widespread, simple and without restrictions. While the scale of reforms has been considerable, it is difficult to see evidence of improved overall governmental effectiveness; for example the International Country Risk Guide (ICRG) rating for Guatemala on bureaucratic quality is unchanged since 2002. However, this is not surprising, given the long expected time span for improved processes to translate into improved outcomes, Given all these considerations, IEG rates efficacy of this objective as substantial.

2. Efficiency. The operation helped achieve important results, including the adoption of Government Finance Statistics (GFS) 2001 international classifications in 2009, with 70 functional classifications incorporated into the 2012 budget. The PEFA had pointed out that despite the creation of an electronic payroll and establishment register with support from the operation, it is not possible to reconcile the payroll with personnel records, and the CGC has identified delays of several months in the updating of payroll records. However, the TTL states that these problems had been largely addressed by 2011.Out of 34 intermediate results indicators (IRI), all were fully achieved except for 3 that were partially achieved: the municipal service system (SERVICIOS GL), which promotes greater efficiency and equity in revenue administration, streamlines the payment of taxes and fees, and puts many services and information on-line, was only implemented in 31 entities (target 50), 2 pilot non-financial specialized audits were performed (target 5), and a single government Internet portal has not yet been established. Many IRIs were added at a late stage by the Government to help clarify to all stakeholders how project achievements had gone beyond expectations. Taking all of this into account, IEG rates efficacy of this objective as substantial.

3 Transparency. The improved SIAF and Government Audit System (SAG) both help improve transparency by producing timely, integrated financial information and audit trails. SIAF/SAG and improved dissemination of data tracked by these systems to the public helps Guatemalans hold public officials to account for the use of public resources. Information is provided on a number of different portals on current and past financial operations, purchase plans, laws and norms. The CGC provides e-learning programs on ethics for government staff, social auditing for the public, and enables on the CGC website reporting on financial wrong doing. In parallel with Bank supported efforts, the civil society portal Consulta Ciudadana offers user-friendly access and interpretation to citizens of financial reports; this would not have been possible if the information were not available to the public. Another welcome sign of government commitment to creating a better society is the Guatemala's creation of the International Commission Against Impunity through an agreement with the UN in 2007, to combat corruption and crime. More recently, in 2011 Guatemala committed to join the Open Government Partnership, a global effort promoting fiscal transparency. The improved transparency may have contributed to a small improvement in controlling corruption (from 1.5 to 2.0 out of 6 from 2002-2012, according to ICRG. The ICR indicates that further progress will require addressing challenges such as reluctance of media to speak out, tolerance of senior officials of lack of compliance to PFM norms, and a culture of impunity. Taking all of this into account, IEG rates efficacy of this objective as substantial.

5. Efficiency:

Though there was no quantitative economic or financial analysis carried out at appraisal, two types of efficiency gains were anticipated: a) fiscal savings-- reduced interest rate fees, savings from elimination of checks for payroll and suppliers, reduced waste and corruption costs from municipalities, reduced administrative costs, reduced corruption from procurement, and higher penalty fees for detected irregularities through enhanced audits; b) efficiency gains-- more focused use of resources, modernization of Treasury and public credit systems, consolidation and expansion of the SIAF, more efficient fiscal transfers, elimination of ghost workers, more effective monitoring and control of financial flows through a strengthened CGC, and better public asset control (Project Appraisal Document, Annex 4 and Project Paper for Additional Financing, p. 12), all leading to potential public savings. Unfortunately, the savings from these efficiency gains are not tracked in the M&E framework (see section 10 below) . However, savings from lower costs and better quality of goods and services purchased, increased tax collection, and better public services quality and delivery at the local and community levels have been documented (Evaluacion de la capacidad de gestión financiera y diagnóstico de los procedimientos de adquisiciones del país (CFAA/CPAR), 2005). In addition, a recent assessment by the MOF estimates that the SIAF system alone has saved about US$60 million since 1998, compared to total disbursed lending of US$69.63 for IFM I, II and III (ICR, p. 11). As SIAF was only one of many well-performing features, it is reasonable to estimate that total savings exceeded the value of the loans, and that efficiency was substantial. The rating could have improved had there been more extensive, quantitative tracking of the efficiency gains expected at project design.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The objective is substantially relevant to addressing challenges of personnel management, tax administration and low tax rates, and weaknesses in public financial management system. The design is substantially relevant, with a convincing causal chain linking funding from the Bank with intended outcomes, and building on achievements of previous operations. Efficacy of achievement of the three key objectives is substantial. The operation has supported improved transparency, adoption of GFS classification, and expansion of SIAF functions, and audit, planning and public investment systems, leading to important advances in almost all aspects of public financial management. The improved processes haven’t yet translated into improved outcome measures, such as ICRG’s rating on bureaucratic quality, but this is not surprising given the long time span required. Based on the available evidence, efficiency can be rated as substantial.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The 15 year period of Bank engagement from preparation of the initial operation to closure of this one gave sufficient time for reforms to take hold. However, three critical risks were highlighted at appraisal that are still present at completion. The first was lack of government commitment to greater transparency and public sector reform. The 2009 PEFA found serious weaknesses in the PFM framework, including inadequate information provided to municipalities, weak link between payroll and personnel, failure of units to comply with regulations and internal control, poor budget classification, and reliance on sole source procurement . While some of these challenges are being addressed, there is less progress on others which is troubling. The second risk highlighted at appraisal is the inability to maintain cadre of trained staff. There has been much progress in this area. MINFIN transferred UNDP project procurement and financial administration to government units, created a Technical Committee to ensure coherence, enhanced training programs, and kept on IT consultants after project completion. However, the risk remains that some of the key, highly marketable IT consultants will not be retained, particularly if fiscal constraints jeopardize the financing of system maintenance and staff compensation packages. A third risk is lack of commitment or ability of municipalities and other stakeholders to implement reforms. As pointed out above, there were implementation delays early in the project in municipalities, and in the CGC, for such reasons. While there now seems to be strong commitment and improved capacity, the ICR points out that there is a risk that individual units might pursue their own priorities, disregarding the integrity of the system. Further, it points out that there is an incentive structure for officials to use public office for personal gains, reputational or otherwise; this could be at odds with increased transparency, which might in turn put the reforms into jeopardy. A related risk was not mentioned at appraisal, nor in the ICR. Despite a peace accord signed in the 1990s, it is estimated that the cost of violence in 2005 was more than twice the combined budget for agriculture, health and education (World Development Report, 2011, p. 5); such a level of conflict can pervert institutional progress, and put fiscal goals at risk.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The design built on lessons from the two previous projects, with a focus on two implementing agencies, oversight by UNDP, and responding to client priorities. The Bank helped mobilize other development partners to promote sufficient counterpart funding. The project had a proper level of ambition-- IEG agrees with the ICR on this point, while the QAG review called the project over ambitious. While the design properly strengthened accountability functions such as CGC and internal audit, it did not fully address the political economy risks stemming from institutional culture.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

Supervision was very capably done, with frequent site visits, and informative Implementation Status Reports (ISRs). The team participated in the PEFA, and worked with counterparts to address identified shortcomings.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The government was committed to broad public sector reform, as seen in the provision of public access to key fiscal information and debt monitoring with summary results publicly available <http://transparencia.minfin.gob.gt/transparencia/>, a multi-year system of budget planning linked to sectoral strategies and plans, and sound tax administration, as pointed out in the 2009 PEFA report. The President and Finance Minister decided to use the PEFA report to address remaining shortcomings, although much more work remains to be done.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:

MOF demonstrated strong project leadership, working well with municipalities and CGC under successive Ministers to fully meet most targets under the project, and to address many of the weaknesses identified by the PEFA report. The CGC performance was uneven, with one Comptroller imprisoned for corruption, while the current one has been a strong champion for reform and the project.

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The objective was well specified, and there was a comprehensive set of measurable indicators presented in the Appraisal Document (Annex 1). The monitoring framework could have been improved with indicators to measure the proposed gains in efficiency and effectiveness foreseen in the PAD (Annex 4).

b. M&E Implementation:

There was comprehensive monitoring beginning with the Implementation Status Report (ISR) of June 9, 2006, and expanded in subsequent ones, with appropriate baselines. The TTL explained that there was no reporting in ISRs on indicators until 2006 because the team was awaiting the provision of a reporting format by Bank management, which was released in 2006. The reporting framework was further developed during the preparation of the additional financing request, and this effort helped facilitate technical and policy dialog with the client. Indicators were again fine tuned in the restructuring of the additional financing request, to align them with what was likely to be achieved. As part of this, the client added a number of indicators to help clarify to all stakeholders how project achievements had gone beyond expectations. Support from the operation helped to improve the financial monitoring capacity of key government units.

a. M&E Utilization:

Monitoring data were used to track progress, and to expand the project scope as appropriate. For example, there were new monitoring indicators added in response to the PEFA, to track efforts to address shortcomings.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

The operation supported improvements in public financial and human resources planning and management that do not have direct effects on safeguard issues. The project was not required to be classified in an environmental or safeguard screening category.

b. Fiduciary Compliance:

Financial management and procurement were initially managed by UNDP, with satisfactory ratings in ISRs. Once these functions were handed over to the MOF, financial management continued at a high standard, with timely quarterly monitoring reports, and annual audit reports that were unqualified. Procurement under government management was rated Moderately Satisfactory, with staff shortages, lack of adherence to procurement plan, incomplete files, and use of some improper processes.

c. Unintended Impacts (positive or negative):

There were no unintended impacts.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Satisfactory
 
Risk to Development Outcome:
Significant
Significant
 
Bank Performance:
Satisfactory
Satisfactory
 
Borrower Performance:
Satisfactory
Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

A key lesson is the value of taking a long term approach to strategic, PFM reform. The World Development Report, 2011 estimates that looking across country experience in the 20th century, it took the fastest 20 reformers 20 years to achieve basic transformation of bureaucratic quality. The last 15 years of PFM reform in Guatemala confirms this long time horizon, showing that while there has been considerable achievement, there is still a large remaining capacity gap to fill. This case shows that even in a difficult situation of a country with high level of conflict, Bank support can be valuable in partnership with committed partners.


14. Assessment Recommended?

Yes
Why?


It would be informative to look at this operation in comparison with others in the region with similar objectives, to see the extent to which the Bank has systematically scaled up approaches that work, and avoided making the same mistakes more than once.


15. Comments on Quality of ICR:

Overall, a comprehensive and highly informative ICR. However, there are discrepancies in amounts of actual loan disbursement, and borrower contribution between amounts in information sheet, ICR basic information, and Annex 1 of the ICR that need to be reconciled. Annex 3 is incomplete, omitting to list the notable achievements.


a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions