Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Economic Capacity Building TA

1. Project Data:   
ICR Review Date Posted:
Project Name:
Economic Capacity Building TA
Project Costs(US $M)
 7.83  8.37
L/C Number:
Loan/Credit (US $M)
 7.54  8.01
Sector Board:
Economic Policy
Cofinancing (US $M)
Board Approval Date
Closing Date
09/30/2007 03/31/2013
Central government administration (90%), Sub-national government administration (10%)
Macroeconomic management (34% - P) Public expenditure financial management and procurement (33% - P) Administrative and civil service reform (33% - P)
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Lev Freinkman
Robert Mark Lacey Lourdes N. Pagaran IEGPS2

2. Project Objectives and Components:

a. Objectives:

    The project development objective (PDO) in the Project Appraisal Document (PAD, page 1) is virtually identical to that in the Development Credit Agreement (DCA, page 12):

    “to assist the Borrower to design and implement a public sector management reform program aimed at improving: (a) budgetary and public expenditure management systems and processes; and (b) civil service and public administration performance.”

b. Were the project objectives/key associated outcome targets revised during implementation?

c. Components:

The project had three components:

(1) Strengthening Public Expenditure Management Processes (appraisal cost – US$5.1 million, actual – US$6.2 million). This component aimed to improve budget preparation, execution and monitoring systems and processes, and included three sub-components (each of which had several smaller sub-divisions):

    • Strengthening medium and short term budget planning (budget formulation and preparation, macro and fiscal policy framework, public expenditure management at the agency level, systems for budget analysis, etc.).
    • Improving budget execution (government financial information systems, procurement reform).
    • Enhancing budgetary oversight (monitoring, evaluation, and internal audit).
(2) Modernizing the Public Administration and Civil Service (appraisal cost - US$2.4 million, actual – US$1.3 million). This component intended to streamline institutional arrangements and build capacity for civil service management. It had the following sub-components:
    • Strengthening civil service management capacity and professionalism (implementation of civil service reform strategy, agency-level efficiency reviews).
    • Establishing a human resource management information system.
    • Rationalizing the civil service wage bill, incentives and employment (compensation policy and severance policy reforms).
    • Strengthening accountability for performance (accountability systems and income and asset declarations).
(3) Supporting project implementation (appraisal cost – US$0.3 million, actual – US$0.8 million). This component intended to provide consulting services, training and equipment to build capacity in the Project Coordination Unit, based in the Ministry of Finance and Economy (MOFE).

A management-endorsed restructuring became effective on April 29, 2009 to take account of the effects of the global financial crisis. Funds were reallocated from the civil service reform component to the public expenditure management component as fiscal management issues became more pressing. The development objectives were not changed.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project cost: Total project cost amounted to US$8.37 million (107% of the appraisal estimate). The difference is due to the appreciation of the Special Drawing Rights (SDR) against the US dollar.

Financing: Total disbursements under the IDA Credit amounted to US$8.01 million, compared to an originally approved amount of US$7.5 million, the difference being due to exchange rate movements. In SDR terms, disbursements totaled to 95% of the originally approved amount, and the rest was canceled at the time of project closure. There were no other sources of external financing.

Borrower contribution: At appraisal it was foreseen that there would be government co-financing of project costs in the amount of US$0.29 million (4% of total project costs). The ICR states (Annex 1) that the actual government contribution was somewhat higher (US$0.36 million) as it expanded in line with the dollar value of IDA’s credit.

Dates: At appraisal, the project was expected to be completed in about 4 years, but the actual implementation took almost 10 years, until March 2013. The project was extended three times at the government’s request to provide more time to complete the ongoing project activities, specifically the implementation of large IT-based management systems. The first extension was granted in April 2007 for two years until September 2009. The second extension was provided in May 2009 for additional 2 years until September 2011, while the decision on the final extension until March 2013 (for a further 18 months) was made in September 2011. The Government’s request for the second extension was linked to a need to provide technical assistance to the newly created (January 2009) National Development and Innovation Committee (this request triggered the project restructuring in April 2009 that was complementary to the second extension). The Government’s request for the third extension explicitly referred to lack of effective project management in the Ministry of Finance as a reason for implementation delays.

3. Relevance of Objectives & Design:

a. Relevance of Objectives:

The project’s public financial management objectives are relevant to the Mongolia Country Partnership Strategy (CPS) for the Fiscal Years 2013-2017, which explicitly places the challenge of a robust fiscal policy at the center of the Bank’s assistance. Under its Outcome 1.2, the CPS commits to “Support the Government in designing and implementing policies and systems for a more robust, equitable and transparent management of public revenues and expenditures.” However, civil service reform, the project’s second technical component, does not feature among the main objectives of the CPS, nor of the Interim Strategy Note for the Fiscal Years 2009-2011. At the time of project approval, the Mongolia Country Assistance Strategy (CAS) for the Fiscal Years1999-2004 placed support to “medium-term macroeconomic stabilization through improved public expenditure management and public administration reform” among its main strategic goals.

The project objectives were also partially relevant to the main goals of the Government’s Economic Growth Support and Poverty Reduction Strategy (approved in July 2003) that emphasized the need to improve effectiveness of public expenditures to reduce poverty and build institutions to improve monitoring of government policies.

b. Relevance of Design:

The Results Framework is presented in Annex 1 of the PAD. With respect to the first project objective (improved budgetary and public expenditure management systems and processes), the framework was sound. It pointed to a reasonable casual chain between project inputs (various investments in IT systems, new laws and regulations, advisory services, and training) across the entire budget cycle, which, if were properly and timely implemented, would had resulted in improved efficiency of the national budget system (outputs) and improved utilization of public resources (outcomes). The weakest point in this logic was underestimation of the time needed to build the capacity of local institutions in order to make them capable of both full utilization of new project-supported systems and implementation of new regulations.

The link between project inputs and outcomes, however, was less evident for the second objective (improved civil service and public administration performance), which was too broad relative to the proposed project interventions that covered a much narrower set of reforms (improvements in the incentive structure, accountability and management arrangements within the civil service). It seems unlikely that even under the best case scenario the implementation of Component 2 could result in broad modernization of public administration (because such an objective would require many additional administrative and sectoral reforms to supplement those aimed at modernization of the civil service).

4. Achievement of Objectives (Efficacy) :

Project objective 1: “To design and implement a fiscal management reform program aimed at improving budgetary and public expenditure management systems and processes.” Modest.

In terms of outputs, the project, though its financing of preparation of new legislation and regulations, advisory services, training programs, etc., helped the Borrower to make some progress toward modernization of several core components of its budget management system. The most important outputs were:

    • Creation of a functional treasury system (GFMIS, launched in 2005) to enhance expenditure controls and improved payment processing.
    • Upgrading of the core fiscal legislation, specifically, adoption of the Integrated Budget Law (IBL, adopted in 2011) and Fiscal Stability Law (FSL, 2009). The latter for the first time introduced institutional conditions for a disciplined budgeting over the commodity cycle. The FSL provided for a set of budget rules that has the potential to strengthen the country’s fiscal policies and budget planning. There were also substantial improvements in the budget calendar, due to IBL requirements.
    • Budget planning reforms: Some progress towards the multi-year expenditure framework (MTEF) was made, but only at the macro level, with no arrangements in place yet for regular preparation and updating of sectoral expenditure strategies. The budget preparation information system (BPIS), which was another single large IT system expected to be delivered by the project, was not operational at closure.
    • Accounting and auditing reforms: the Ministry of Finance has established an internal audit department, but is still in the process of developing a methodology for internal audit of government accounts. While the timing of external budget audits has been improved, the government had not yet started to publicize their findings on a regular basis.
    • Procurement: The country’s Procurement Law was upgraded and necessary supporting regulations were adopted, which has increased the operational efficiency of the national procurement system. But implementation progress has been slow due to slow build-up of local capacity (despite project support) and exemptions from the competitive bidding requirements granted by the government.
    • Investment planning: Project appraisals are mandated in the IBL, but this requirement is not effectively enforced.

In terms of Outcomes, progress was limited. At project closure, many new systems remained under-utilized, and new laws and regulations were not regularly enforced by the government. This was due, in part, to capacity bottlenecks, and in some cases to lack of political will to move toward higher budget transparency. There were, however, improvements in budget credibility and discipline, as well as a decline in budget arrears. This progress was attributed to the introduction of the Government Financial Management Information System (GFMIS), which, in the opinion of the Bank, has been the single major reform achievement in the PFM area in Mongolia over the last decade. However, as of late 2013, not all GFMIS functional components had been made operational and the system remained underutilized. As a result, GFMIS operations had not, by project closure, led to any significant improvements in the quality of budget reporting or availability of budgetary information.
Moreover, Mongolia’s 2013 national budget, the first one expected to be prepared, approved and executed in line with the new requirements under the FSL, did not comply with the law, since the budgeted consolidated deficit exceeded the stipulated threshold.

Project objective 2: “To design and implement a public sector management reform program aimed at improving: civil service and public administration performance.” Negligible.

There was less government’s ownership of the civil service reform (CSR) agenda than of public financial management enhancements. The project, therefore, had less traction in this area. Key project outputs included amendments to the Civil Service Law (adopted in 2008) and approval (in 2007) of the Medium-Term Civil Service Reform Strategy. Both were prepared with substantial inputs funded by the project.

The original project design provided for supporting the Borrower in the implementation of this strategy. However, there was little drive to implement it from the government side. In response to this lack of ownership, the Bank supervision team decided to (i) downsize the respective project component through project restructuring, and (ii) to concentrate efforts under the restructured CSR component on implementation of the Human Resource Management Information System (HRMIS). However, the latter task remained incomplete by the project closure despite much effort and additional project extensions. The only outputs generated included detailed business and technical specifications and demonstration software for the HRMIS. According to the ICR, those outputs may provide a basis for the future development of the HRMIS.

No civil service reform outcomes were achieved.

5. Efficiency:

No attempt was made to provide a quantitative measure of the degree of efficiency with which the project utilized IDA funds. However, there are indications of very low efficiency. Almost 40% of all project funds were spent on activities which resulted in practically no development outcomes being achieved (civil service reform, the development of a medium-term expenditure framework, and investment planning). A further indicator of low efficiency is the time taken to implement the project – 9.75 years against an appraisal estimate of 4.25 years. The main reasons for these delays, according to the project documents, relate to capacity constraints (for example, in preparing timely and quality tender documents for complex IT systems and then effectively supervising the contractors) and to weak ownership (leading, for example, to lack of implementation of the agreed civil service reforms).

Efficiency is rated negligible.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:

Rate Available?
Point Value
ICR estimate:

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Although the project’s objectives were substantially relevant, its design did not adequately account for country conditions. An overly complex project with too broad a coverage and excessively ambitious goals did not fit well with local implementation capabilities. While some outputs related to public financial management may provide groundwork for future reform efforts, achievement of the development objectives was modest with respect to financial management, and negligible with regard to civil service reform. Efficiency is rated negligible, reflecting major operational and administrative inefficiencies, lower outputs and no development outcomes for activities accounting for 40% of project financing, and the fact that no attempt was made to quantify the efficiency with which IDA resources were utilized.

a. Outcome Rating: Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

Mongolia’s progress toward more responsible fiscal management remains incomplete and has to be consolidated. Recent failure to comply with the requirements of the Fiscal Stability Law as well as to reform the system of universal social transfers (to make it more poverty-focused) indicates that there is still no established consensus within the country’s elite to support a sustainable fiscal strategy, while populist sentiments are quite influential. This suggests a significant risk of potential erosion of progress achieved under the project. Such a risk is further augmented by currently high prices of Mongolia’s main commodity exports, which tends to undermine the local demand for efficient expenditure management.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The project intended to address the top priority needs of the client and do it in a way that would ensure continuity of the engagement. It was designed as a follow-up to the earlier Fiscal Technical Assistance Project (FTAP, which closed in 2005) and had many similarities with that project in terms of objectives, design, coverage, and core implementation arrangements. The Bank learned a key lesson from similar projects in other low capacity countries, namely the need for long-term engagement. Preparation was also informed by the findings of a Public Expenditure and Financial Management Review, prepared by the Bank in 2002 in collaboration with other donors.

However, design had major shortcomings, being too broad and ambitious. The project, over its original life time of four years, aimed to secure major progress across the whole spectrum of the national Public Financial Management (PFM) systems, backed up by quick installation of large and sophisticated Information Technology (IT) systems. The literature on governance and PFM reforms provides warnings against attempts to move rapidly to the best international standards of PFM practice in lower income countries. In particular, the lessons from the preceding FTAP were not fully taken into account. These lessons, as spelled in the ICR Review for the FTAP, included inter alia a need for the Bank to (a) conduct a realistic assessment of the Borrower’s capacity during appraisal, and (b) to pay more attention to the proposed time frame for project implementation. Nor did the design of the project under review incorporate the government’s implementation constraints in a realistic manner. As such, the project design did not reflect adequately one of the earlier concerns of Mongolia Country Assistance Evaluation (CAE), prepared by IEG in 2002 (and thus available by the time of project approval), that local implementation capacity remained a major constraint for utilization of the Bank’s support. The critical local capacity gap, not fully addressed in the design, relates, in particular, to insufficient management skills to handle the implementation of large IT-based systems. An alternative design, focusing on fewer reforms, and consolidation of the less sophisticated and less-IT-dependent systems first, would have been more appropriate.

The design of the civil service reform component was not based on in-depth analytical work (the major report on this topic was done by the Bank only in 2009) and, as noted in the ICR, it did not identify local champions for a comprehensive civil service reform, who would be willing to invest their time and effort in project implementation in the highly segmented institutional set-up. Moreover, the country-specific risk to reform ownership was not adequately reflected in the project design.

The cyclical nature of PFM in Mongolia was not sufficiently taken into account. During the commodity boom of 2003-07, in the environment of rapidly growing public spending, the need for expenditure rationalization appeared less pressing in Mongolia, which naturally eroded the political will to push towards more PFM reforms.

Quality-at-Entry Rating: Unsatisfactory

b. Quality of supervision:

There were major shortcomings with the Bank’s supervision. First, the team introduced a change to the original monitoring arrangements, which impeded proper monitoring of implementation progress – it switched from the set of result indicators agreed in the PAD to a narrower/weaker set of indicators. The full set of monitoring information, consistent with that in the PAD, was collected only once, at the time of project closure, which suggests that a considerable amount of information on outcomes was available during the project life, but it was not fully collected and utilized in the course of supervision (See Section 10 below). Second, the team's assessments of implementation progress and progress towards reaching development objectives lacked candor. The Satisfactory rating of implementation progress was maintained in supervision reports until the time of restructuring in April 2009, despite the procurement problems and delays with major contracts (including two unsuccessful procurements for the largest IT contracts) that had become apparent much earlier. Between 2009 and early 2013, the project was rated as Moderately Satisfactory in terms of progress towards achievement of development objectives, although only modest progress was apparent for the first objective, and virtually none for the second. Third, the downsizing of the non-performing civil service reform (CSR) component was not undertaken until 2009, although the major ownership issues on the Borrower side should have been identified and addressed earlier. According to the project team, the client was not ready to downsize the CSR component before the severe economic crisis hit Mongolia in 2009, but the Borrower’s unwillingness to implement the component was clearly evident prior to the crisis, as there was very little activity in the first four years of implementation. Fourth, as the ICR acknowledges, the supervision team was not equipped for extended periods with the IT skills necessary to help the Borrower with critical procurement and implementation of large IT systems.

According to the ICR, no formal mid-term review was held, since the discussions held at the time of the 2009 restructuring de facto constituted such a review. However, the project files point to a mid-term review discussion held in early 2007, during which the parties agreed on the initial project extension (for two years) and the steps to strengthen capabilities of the Government Service Council (GSC) in the attempt to re-energize implementation of the CSR component.

Quality of Supervision Rating: Unsatisfactory

Overall Bank Performance Rating: Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

Combined assessment of Government/Implementing Agency Performance:

One reason for the project’s over-ambitious design was that the government itself (with advice from development partners) had, in 2002, declared a far-reaching set of reform targets inspired by its then new Organic Fiscal Law (Public Sector Financial Management Law, PSMFL). The PSMFL provided for the introduction of advanced public financial management practices in Mongolia, including the adoption of output based budgeting and accrual budgeting and accounting. Thus the project design was aligned with the government’s own reform strategy, but in the course of implementation this strategy proved to be unrealistic and over ambitious.

The main implementing agencies according to the original project design were the Ministry of Finance and the Governance Service Council (GSC). The National Development and Innovation Committee (NDIC) became another key implementation agency after the project restructuring.

The main issues affecting the project performance at the government level included: (i) lack of ownership for the civil service reform (Component 2), and (ii) insufficient coordination between different government agencies, for example, between the Ministry of Finance and the NDIC with respect to the implementation of new systems for investment project evaluation. The primary responsibility for project implementation was given to the Ministry of Finance, which hosted the Project Management Unit (PMU). The PAD reports that there was a provision for a central Project Steering Committee with the responsibility for strategic guidance on project developments and monitoring, but there is no reference to the actual establishment of such a Committee in project documents. According to the ICR, a project-specific steering committee with broad representation was set up at a late stage in the life of the project in the attempt to accelerate implementation of the Human Resource Management Information System.

Combined with insufficient implementation capacity in key agencies, these factors were a major source of implementation delays. In addition, it took more than seven months for the project to become effective. The primary reason for effectiveness delay, according to the project team, was the need to get the project agreement ratified by the national Parliament, which was a lengthy administrative and legal procedure that was causing delays for many Bank operations.

Implementation was seriously affected by inadequate project management capabilities in several government agencies, related especially to management and supervision of large ICT contracts.

Moreover weak management of large procurements resulted in major fiduciary issues -- two investigations by the Bank’s INT Unit, debarment of the contractor, and contract termination (see Section 11 below).

Counterpart funding amounted to US$0.36 million (against US$0.29 million agreed in the PAD).

Government Performance Rating: Unsatisfactory

b. Implementing Agency Performance:

Implementing Agency Performance Rating: Not Applicable

Overall Borrower Performance Rating: Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

PDO indicators in the main text of the PAD (p. 3) differ from those in the results framework (Annex 1). Ten indicators in the main text reflect well the objectives activities under the project, although several of them are process or output-oriented. Annex 1 presents an expanded list of 26 output indicators, which includes all the original 10 from the main text. If either of these two sets of indicators had been used, progress against the project PDOs would have been reasonably well measured. However, this was not done (see 10b below).

The PAD assigned responsibility for collecting the monitoring data to the Project Management Unit (based in the Ministry of Finance) under the supervision of the Project Steering Committee. The PAD also outlined a reasonable strategy for collecting necessary data. However, there were no baseline data presented.

b. M&E Implementation:

The original monitoring strategy was not applied during implementation. During implementation, the team monitored quite a different, much narrower set of six result indicators. Four out of those six were substantively similar to the output indicators from the original list in the PAD Annex 1. But generally the result framework actually used by the team seems to have been weaker than the one in the PAD because its coverage was narrower (with a smaller number of monitoring indicators), and it was excessively focused on the implementation of large IT systems funded under the project. This change, according to the ICR, was motivated by a need to “sharpen the M&E in line with general improvements to World Bank project M&E practices”, but in practice the substitution weakened the link between project objectives and monitoring indicators. The collection of full set of original results indicators was undertaken only once during the project’s life, at the time of the final supervision report.

The change to the M&E framework was formally agreed with the Borrower and approved by Bank Management in May 2010, six years after the de facto replacement of the original results framework. The change in the framework was not made part of the official project restructuring.

a. M&E Utilization:

Supervision relied on the narrow set of monitoring indicators focusing largely on project outputs, not outcomes. As the final supervision report suggests, more information on outcomes was available, but not collected.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

There were no safeguards policies triggered by this project which was classified as Category “C” for Environmental Assessment purposes.

b. Fiduciary Compliance:

Financial Management. The ICR and supervision reports present two cases of fiduciary non-compliance. On both occasions, the Bank’s Integrity Unit conducted an investigation. In the first case, the investigation (launched in 2008) had revealed the fraudulent behavior of the contractor working on the Budget Planning Project that eventually led to debarment and contract’s termination. In the second case, where the INT investigation remains incomplete, there was a complaint over suspected conflict of interest at the Project Coordination Unit, where a certain contractor might have been favored in exchange for favors given to a certain consultant. At the time of writing, no further information is available about these investigations.

The ICR provides no discussion of fiduciary compliance apart from the issues under INT investigation. According to the project team, the project’s accounts were externally audited on an annual basis and were unqualified.

Procurement. The ICR contains no systematic discussion of procurement issues. It is mentioned that there were difficulties with large scale IT procurements, but few details are provided. It is not stated whether there were any cases of misprocurement.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:

12. Ratings:

IEG Review
Reason for Disagreement/Comments
Risk to Development Outcome:
Bank Performance:
Moderately Unsatisfactory
There were major shortcomings in the quality of supervision, including weak monitoring of project progress, delayed restructuring, and lack of candor in most project ratings. 
Borrower Performance:
Moderately Unsatisfactory
There were major shortcomings in Borrower performance, including weak coordination, insufficient ownership of civil service reforms, and inadequate efforts to build implementation capacity.  
Quality of ICR:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The following lessons are taken from the ICR with some adaptation of language:
    • In an environment of low absorptive capacity, an implementation strategy that relies on large scale consulting contracts is risky, and if followed, requires a significant amount of implementation support from the Bank team. An alternative strategy of using more, but smaller, contracts might be easier to manage.
    • Continuity of Bank’s assistance, based on the long-term engagement with the client, is critical to secure robust institutional development and consolidate reform gains.

The following lesson is drawn by IEG:
    • Project teams preparing technical assistance operations with institutional strengthening objectives, may need guidance from Management in applying relevant lessons from similar projects undertaken elsewhere. This would help to prevent an observable repetition of excessively broad project coverage, over- ambitious objectives and an unrealistic implementation pace. This repetition frequently occurs despite the abundant evidence that such strategies are usually unsuccessful in an environment of low capacity and insufficient client’s ownership.

14. Assessment Recommended?


15. Comments on Quality of ICR:

Overall, the ICR provides a credible, informative, and candid presentation of project preparation and implementation history. Based on analysis of the relevant literature on the public finance management, it provides a number of good insights with respect to the project’s design weaknesses. The outlined lessons, while not particularly new, are quite relevant and well-grounded in the project’s experience. However, from the evaluation perspective, the ICR is not sufficiently critical of Bank’s performance at project supervision. In particular, it does not discuss the lack of candor in supervision ratings.

Discussion of fiduciary compliance is inadequate, perhaps in part reflecting the fact that INT investigations were still ongoing at the time of ICR preparation.

There are several factual inaccuracies. First, and most importantly, there were imprecisions regarding the changes made to the M&E framework. The ICR states (p.3) that the performance indicators were modified in 2009. In practice, the project team had switched to the indicators much earlier, not later than in 2005, although this change was only formally endorsed in Fiscal Year 2010. Second, contrary to what is stated in the ICR, the mid-term review took place in 2007. Third, costs of project supervision are substantially under-reported in Annex 3, since they are provided only for the period up to Fiscal Year 2008, that is for the first half of implementation. Fourth the data on actual disbursements in Annex 1 are incorrect -- the correct numbers, expressed in US dollars, are available in the Bank’s Operations Portal and are consistent with the SDR amounts reported in the ICR.

a. Quality of ICR Rating: Satisfactory

© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions