Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Second Economic Governance Technical Assistance Project


  
1. Project Data:   
ICR Review Date Posted:
06/19/2012   
Country:
Haiti
PROJ ID:
P095371
Appraisal
Actual
Project Name:
Second Economic Governance Technical Assistance Project
Project Costs(US $M)
 2.00  0.475
L/C Number:
CH237
Loan/Credit (US $M)
 2.00  0.475
Sector Board:
Public Sector Governance
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  06/20/2006
 
 
Closing Date
06/30/2009 06/30/2011
Sector(s):
Central government administration (100%)
Theme(s):
Other accountability/anti-corruption (25% - P) Administrative and civil service reform (25% - P) Public expenditure financial management and procurement (24% - P) Participation and civic engagement (13% - S) Law reform (13% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Jiro Tominaga
Michael R. Lav Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The original development objective as indicated in the Project Appraisal Document (PAD) is to further assist the Government of Haiti (Government) in strengthening its institutional capacity in the areas of public sector resource management and improved responsiveness to citizen needs and demands. (Section 3.a., page 12, PAD)

    The Financing Agreement indicates the objective to be to assist the Recipient in strengthening further its institutional capacity in the area of public sector resource management and achieving improved responsiveness to its citizens' needs and demands.

    The PDO was revised in October 2009. The revised objective is to assist the Government in strengthening its capacity for: (a) budget formulation and execution and; (ii) budget monitoring and control.

    This review assesses the achievements against the original PDO as well as the revised PDO.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 10/30/2009

c. Components:

At the time of appraisal, the project consisted of five components.

Component 1. Financial Management (appraisal cost $0.7 million): It aims at the development of a capacity to produce and execute a budget, track its implementation, and reduce irregular uses and expenditures.

1.1. Budgeting: Continue EGTAC I's focus on design and implementation of basic budgetary procedures and functions within the MEF and the Ministry of Planning and the Coordination of External Assistance (MPCE) as established by the new Organic Law. The goal is to move the process beyond the EGTAG I objective of incorporating all central government expenditures (including donor contributions) in the official budget toward a more program-oriented, results-linked format. Also to finance an evaluation of the initial results of the new procedures, identify areas for their modification or strengthening, and support technical assistance, training and some equipment purchases.

1.2. External Control: The LICUS grant funding for the Cour Supérieure des Comptes et du Contentieux Administratif (CSCCA) supported class-room and on-the-job training of auditors in their new ex-post control function and the drafting of an updated Organic Law. Under EGTAG I, the CSCCA's initial experience with ex-post auditing is being evaluated, and further personnel, equipment, budgetary, and training needs identified, resulting in a plan for upgrading the agency's performance over the following two to three years. EGTAG II is to support implementation of and further adjustments to this plan. Work will be coordinated with assistance provided by the IDB in the area of ex-post internal control.

1.3. Procurement: Continue the support of the newly created Commission Nationale des Marchés Publics (CNMP) for its transformation into the permanent body mandated by law, via strengthening of its own capacity and the development of training programs for sector procurement staff in standardized procurement procedures. Also assess and build on progress made in the initial development of standardized bidding documents, publication of information of bids and contracts, and the education of potential bidders and the general public on procurement processes. All operations to be closely coordinated with related IDB technical assistance to key sector ministries.

Component 2. Human Resource Development (appraisal cost $0.5 million): Under EGTAG I, assistance was limited to upgrading training facilities and supporting for planning the development of a human resource system. EGTAG II is to advance these processes through the creation of organizational structures, the development of basic human resource management tools, and the training of central and sector staff in their uses. Also, finance studies related to the improvement of the public administration's legal framework.

2.1. Strengthening of Civil Service Norms and Regulations: Evaluate the two new decree-laws on the civil service and central administration organization, and provide technical assistance for their implementation or possible modification. In addition, support developing procedures for merit selection and evaluation of employee performance, as well as advances in job classification and rationalization of salaries as a means of establishing performance incentives.

2.2. Improving Human Resource Management Information: Provide support to the Prime Minister's Human Resources Office in building its capacity to monitor human resource needs, policy, and programs; to develop ties with and strengthen sector human resource departments, and to help other agencies develop plans for meeting their current and future personnel requirements. Expand EGTAG I's initial support to finance the creation of a single public employees' data base and a uniform set of personnel records.

2.3. Building Organizational Capacity for Human Resource Management and Training. Assist the Human Resource Unit and other agencies to design and spearhead a longer-term civil service reform. Provide technical assistance to help the Unit develop a staged plan for putting the new requirements into effect and for creating mechanisms to oversee their implementation. Also, improve the Unit's ability to plan training, and the capacity of public sector training institutions to deliver the training needed.

Component 3. Development of MEF (Office of Economic Studies) and MPCE's Capacity for Strategic Monitoring of Sector Policy via Sector Ministries' Tracking of Service Delivery (appraisal cost $0.3 million)

3.1. Support to the MEF Office of Economic Studies in developing a capacity to link recurrent and investment budgets to higher order government priorities: Provide technical assistance to the office in its initial development of a systemic approach to linking policy priorities, budgeting, and results tracking.

3.2. Selection of key agencies for the pilot exercise and evaluation of their existing ability to track service delivery. As part of the selection process and as an input to further work, conduct a thorough diagnosis of candidate agencies' current practices, needs, and capabilities in monitoring service delivery.

3.3. Technical and material assistance to the selected agencies in developing procedures for tracking service delivery, whether done by direct employees or through private providers.

Component 4. Anti-corruption and Civil Society Engagement (appraisal cost $0.25 million): Help develop the Unité de Lutte Contre la Corruption's (ULCC) programming capacity, its skills and techniques for determining institutional vulnerabilities (corruption diagnostics) and the design of their remedies, and provide assistance in drafting and supervising implementation of improved laws (notably, a freedom of information law). Also, provide assistance to improve their performance based on findings of an EGTAG I-funded evaluation of the civil society monitoring programs.

Component 5. Communication, Donor Coordination, and Project Management (appraisal cost $0.19 million): Provide technical and material assistance to the Project Coordination Unit (PCU); and continue to provide support for communications activities on the overall economic governance reforms. This includes implementing a communications strategy and promoting two-way communications between the government and stakeholders to build consensus on the programs, increase understanding of their contents and objectives, and promote input from stakeholders.

The appraisal cost includes an allocation of $60,000 for activities to be programmed.

The project went through a restructuring involving revisions in the PDO (as indicated above), outcome indicators, and reorganization of project components. The restructuring was approved by the Board in October 2009. The reorganization of project components reduced the number components from five to the following three components.

New component 1. Strengthen Budget Formulation and Execution (allocated amount at restructuring $1.32 million, actual $0.268 million) includes the following activities.

    • Modified original sub-component 1.1. Budgeting: Activities were scaled up to include debt management system related activities and capacity building activities to support budget reform
    • Modified original sub-component 1.3. Procurement: Activities were maintained and scaled up to include a dissemination and communication strategy.
    • Modified component 2. Human Resource Development: Reduction of scope. The revised activities include: (i) purchase of equipment; (ii) training and capacity building on the database; (iii) a study on non-permanent employees; and (iv) training programs in collaboration with CNMP and MEF.
    • Original component 3. Development of MEF (Office of Economic Studies) and MPCE's Capacity for Strategic Monitoring of Sector Policy via Sector Ministries' Tracking of Service Delivery without any changes.

New Component 2. Strengthening Budget Monitoring and Control (allocated amount at restructuring $0.48 million, actual $0.053 million) includes the following activities.
    • Modified original sub-component 1.2. External Control: Revised scope includes the design, implementation of a strategic plan, the finalization of the Organic Law and capacity building activities to strengthen performance of the CSCCA. The new component 2 also includes new activity related to internal control--capacity building activities for the newly created General Finance Inspectorate
    • Modified original component 4. Anti-corruption and Civil Society Engagement: The activities related to the ULCC was scaled up to include a Youth Forum on Corruption. The activities related to civil society for anti-corruption were scaled down and limited to a stock taking study on Civil Society mechanisms and capacity building activities for Parliamentarians.

New component 3. Communication and Project Management (allocated amount at restructuring $0.2 million, actual $0.154 million) includes the following activities.
    • Modified original component 5. Communication, Donor Coordination, and Project Management: Scaled up to include M&E capacity building activities and the hiring of a deputy accountant. Database of governance activities was dropped.

The amount left to be programmed was allocated to the new Components I and II.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

The project cost was $2 million financed by IDA grant for $2 million. The project was appraised in April 2006, approved by the Board in June 2006, and became effective in August 2007. The project was restructured and the closing date was extended for 24 months in October 2009 and closed two years after the original closing date in June 2011.

    Project implementation was delayed due to several reasons. First, there was uncertainty whether this grant would require ratification by the National Assembly. The grant became effective 14 months after approval following an agreement by the Government and the National Assembly that exempted this grant from the requirement for ratification by the Assembly. Second, the delay in EGTAG I affected the implementation of EGTAG II as the activities under these grants were envisaged to be linked closely. Third, a series of unanticipated events affected the implementation including hurricanes and tropical storms in Aug.-Sep. 2008, an earthquake in 2010, a serious accident involving the Bank TTL (May 2010), and the murder of the project Coordinator a few weeks later.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

    The objectives were relevant to the Interim Cooperation Framework (ICF) which was the foundation upon which the medium-term National Growth and Poverty Reduction Strategy Paper was to be prepared. The project has particular relevance to one of the pillars of the ICF, strengthening economic governance and contributing to institutional development. The project objectives were also relevant to the Bank's Transitional Support Strategy (TSS) for FY2005-6. The project was consistent with the TSS' focus on restoring credibility in institutions, strengthening economic governance and promoting inclusion and consensus on development priorities. The relevance of the issues remain in the latest Interim Strategy Note for CY 2012. The ISN's cross cutting theme, strengthening governance, aims to promote sound policies in economic governance and capacity development in key areas for reconstruction. Based on the above, the relevance of original objectives is substantial.
    The revision of the PDO narrowed the scope to focus on: (i) budget formulation and execution; and (ii) budget monitoring and control. These objectives remain relevant in the context of the governance theme of the new ISN.

b. Relevance of Design:

The project implementation suffered from complex and ambitious design in relation to capacity constraints and financing envelop. The envisaged activities covered a variety of areas ranging from budget, control, procurement, civil service human resource management, monitoring, anti-corruption and strategic communications. It was also expected that various components are sequenced with the activities in EGTAG I and coordinated with relevant development partners. However, the capacity of the PCU to manage several projects financed by the IDA and IDB was constrained. The Government's ownership of the proposed reforms at the time of the appraisal appears to be questionable as raised in the 2008 QAG report. Given these, the relevance of the original design is modest.

The revised design reduced the complexity associated with the original design. However, the project still involved over ten counterpart institutions and comprised more than 40 procurement processes, including several small contracts for individual consultants. The strain on the implementation capacity remained. The relevance of the revised design, therefore, is also modest.


4. Achievement of Objectives (Efficacy) :

The original development objective was to strengthen the GoH institutional capacity in the areas of public sector resource management and improved responsiveness to citizen needs and demands. The PDO indicators established in the PAD are: (i) percentage of non-salary current public expenditures going through comptes courants; (ii) inclusion of external financing (disbursements and expenditures) in public budget; and (iii) number of ministries developing budgets in program and results format. Indicators (i) and (ii) are reported as part of EGTAG I's PDO indicators. According to the EGTAG I ICR, indicator (i) reached 3 percent, exceeding the target of less than 8 percent, in December 2009; and indicator (ii) achieved the target of 100 percent. Indicator (iii) was not reported in any ISRs, so achievement against this indicator is not known. While indicators (i) and (ii) achieved the targets, the contribution of EGTAG II is doubtful as these targets were achieved as part of EGTAG I activities. The disbursement made before the PDO was revised was also limited, so it is difficult to assume the activities under this project had any impact to these results. Therefore, the efficacy of the project with regard to the original PDO was negligible.

The revised PDO has two parts--budget formulation and execution; and budget monitoring and control. The achievement of the first part of the revised PDO was to be measured by: PEFA indicator 6 (comprehensiveness of information included in budget documentation); and PEFA indicator 19 (i) (use of competition in award of contracts).

Based on the 2011 assessment of PEFA indicators by the Bank team, neither of the targets were achieved (Note: A PEFA assessment financed by the EU is being prepared and it has not been published at the time of the preparation of the ICR and this ICR Review). The baseline for PEFA indicator 6 was C, meaning that budget documentation fulfilled 3-4 out of the 9 information benchmarks. The target was to achieve an improvement to C+, but the status was considered to remain as C in the Bank team's assessment. As for PEFA indicator 19(i), the target was to achieve C (baseline Not Rated), but the Bank team's assessment considers the status is D.

As for the second part of the revised PDO, the indicators are PEFA indicator 26 (ii) (timeliness of submission of audit reports to legislature; and increased understanding of corruption practices as measured by studies' dissemination seminars exit surveys. Again, neither of the indicators were achieved. The revised PDO aimed to obtain C in PEFA indicator 26(ii) (baseline D+). There is a sign of improvement in the timeliness of the submission of public accountants' annual audits. However, the Bank team's assessment did not rate this particular indicator--the 08/09 audit report could not be transmitted to Parliament with the budget as is the norm, since the FY10-11 budget was adopted by decree due to the earthquake. As for the understanding of corruption practices, the extent of progress is not known as the survey was not undertaken.

Due to inability to reach any of the targets for PDO indicators, the efficacy of the achievement of the revised PDO is rated negligible.


5. Efficiency:

The project was not able to contribute to the achievement of the development objectives set at the time of appraisal and restructuring. The efficiency in achieving the objectives thus is rated negligible for both the original and revised PDO.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Both the original and revised development objectives of the project were relevant, and aligned with the Bank's and Government's strategies. However, the relevance of design is modest given its complexity and ambitiousness. Efficacy is negligible for both the original and revised PDOs. Efficiency is also rated negligible throughout the project. The project is rated as unsatisfactory against both the original and the revised PDO. The overall rating, therefore, is unsatisfactory.


a. Outcome Rating: Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:


The review agrees with the ICR's risk rating due to the reconstruction effort being undertaken after the devastating earthquake. Moreover, the risk of downturn in the global economy remains. The slow down of the economy could have negative impact on Haiti through reduced remittances, donor funding and investments. Finally, institutional capacity of the Government remains a risk for achieving development outcomes as pointed out in the latest Interim Strategy Note (for CY 2012).

a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:

The objectives of the operation were relevant and aligned with the overall thrust to improve economic governance. The preparation took place in collaboration with other development partners. Lessons from previous operations (EGTAG I, LICUS Task Force Report, OED's CAE for Haiti) were consulted during preparation. However, as pointed out earlier, the project design was overly complex and ambitious given the Borrower's and Implementing Agency's capacity. The overlap in implementation period of the precursor project, EGTAG I, added to the complexity. The realistic targets and the baseline to track the progress in achieving the PDOs were not set at the time of appraisal. The quality at entry is thus rated as moderately unsatisfactory.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

The task team made considerable efforts in sustaining the dialogue with the client even during the most difficult time of project implementation. However, the ISRs do not provide information in sufficient detail on the challenges in project implementation. The development objective and overall implementation progress ratings in the ISRs remained moderately satisfactory until April 2010, slightly more than a year before the closing, which appears to be high given the shortfalls in progress at that time. The quality of supervision is thus rated as moderately unsatisfactory.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government remained committed to the development objectives of the project in a difficult time after the earthquake in 2010 as demonstrated in its strategy for reconstruction and development--the Action Plan for Haiti Recovery. Restructuring was carried out with strong commitment from the Government. While more pro-active and concerted actions on the side of the Government to facilitate the implementation of the project, government performance is rated moderately satisfactory given the circumstances.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

The Project Coordination Unit (PCU) was tasked to handle activities related to financial management, oversight of procurement, review and channelling of documentation, ensure compliance with IDA and GOH regulations, and ensure inter-agency communication and coordination. However, the PCU was not able to facilitate the effective implementation of the project. It also had shortfalls in the submission of the Interim Financial Reports and the updated operation manual after the restructuring.

    The direct responsibility for project implementation rests on executing entities, whose performance varied. Some progress was made in the implementation of a debt management system or the definition of a strategy for economic recovery by MEF. However, limited progress was made for other agencies. The performance of the implementing agencies is therefore moderately unsatisfactory.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The original project objectives aimed to strengthen institutional capacity in a wide variety of areas of public sector resource management and responsiveness to citizen needs and demands. However, the three outcome indicators cover these objectives partially. They are all associated with the implementation of the budget rules. They include: (i) percentage of non-salary current public expenditures going through comptes courrants; (ii) inclusion of external financing in public budget; and (iii) number of ministries development budgets in program and results format.

    The outcome indictors for the revised PDO were based primarily on PEFA indicators, but two out of four indicators did not have baseline. The intermediate outcome indicators were not formally revised after the restructuring.

b. M&E Implementation:

The team had difficulty in collecting reliable data to monitor the progress. The series of ISRs under the original PDO were not able to report the current status for many of the indicators.

a. M&E Utilization:

As some of the indicators were common with the EGTAG I, data produced in EGTAG I were informed the progress in EGTAG II to certain extent before the restructuring. Given the difficulty in collecting reliable data, it is unlikely to assume that the M&E was utilized effectively to guide the project implementation.

M&E Quality Rating: Negligible

11. Other Issues:

a. Safeguards:
Safeguard is not applicable.

b. Fiduciary Compliance:

There were no systematic fiduciary issues reported.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Unsatisfactory
Unsatisfactory
 
Risk to Development Outcome:
High
High
 
Bank Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

This review supports the lessons provided in the ICR, with particular emphasis on the value of keeping the design both simple and focused. The project suffered from an ambitious design which aims to cover the economic governance issues comprehensively. Given the interlinked and cross-cutting nature of the issues related to governance, the attempt to cover a variety of areas is understandable. However, in an environment with constrained capacity, identifying an appropriate entry point for reform is important (though challenging).


14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is candid in pointing out the weaknesses of the project design and implementation. A question remains why this project, which aimed to expand the gains from EGTAG I, went ahead while EGTAG I was still under full implementation. In this context, it would have been useful to record what kind of assumptions were made at the time of appraisal, and how they led to the decision on the original project design and the sequencing with EGTAG I.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions