Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Emergency Development Policy Operation


  
1. Project Data:   
ICR Review Date Posted:
11/21/2012   
Country:
Haiti
Is this review for a Programmatic Series?
 No
First Project ID:
P118239
Appraisal
Actual
Project Name:
Emergency Development Policy Operation
Project Costs(US $M)
 55  55.58
L/C Number:
Loan/Credit (US $M)
 30  30.58
Sector Board:
Economic Policy
Cofinancing (US $M)
 25  25
Cofinanciers:
Haitian reconstruction Fund
Board Approval Date
  08/05/2010
 
 
Closing Date
09/30/2011 09/30/2011
Sector(s):
Central government administration (75%), Power (25%)
Theme(s):
Public expenditure financial management and procurement (88%) Other accountability/anti-corruption (12%)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Florence M. Charlier
Fareed M. A. Hassan Navin Girishankar IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The objective of the operation is to strengthen and implement the legal and regulatory frameworks and systems to create the conditions for accountability and transparency in the context of reconstruction (Section V Program Document and the summary section). This would be achieved through four specific objectives : (i) enhancing the transparency and monitoring of financial transfers in the electricity sector, (ii) reinstating budget controls, (iii) advancing transparency in public affairs through enforcement of the dispositions of the declaration of asset law, (iv) reinforcing procurement regulation and enhancing transparency in procurement practices (source: operation policy matrix p. 64-66).

    IEG notes that the description of the second and third objectives in the Program Document (section 5-p34) differed slightly -- but not materially -- from the description provided in the summary section of the Program Document. The Program Document referred to "(ii) reinstating the external and internal control audit functions and (iii) strengthening the monitoring mechanisms of the declaration of assets," while the summary section (p.i.) referred to "(ii) reinstating budget controls and the external and internal audit processes; (iii) improving observance of anticorruption measures through better enforcement of the Declaration of Assets Law."

b. If this is a single DPL operation (not part of a series), were the project objectives/key associated outcome targets revised during implementation?
No

c. Policy Areas:

The proposed operation supported key policy actions under four broad policy areas, two of which relate to public sector governance -- anti-corruption and procurement.

(i) Transparency in the management of the transfers to the electricity sector: This area aimed to strengthen the government's capacity to manage public resources efficiently and transparently in a context where transfers to the electricity sector are expected to increase given the impact of the earthquake on the electricity sector and the need to expand electricity coverage. Measures included the elaboration of a monitoring mechanism and the regular publications of payments made by the state to public electricity company (EDH) and to each independent power providers (IPP).

(ii) Public finance management: This area sought to re-establish budget controls and external and internal audit processes. Indeed, progress achieved in terms of external controls was slowed down by the impact of the earthquake on the superior Court of Audits and Administrative Disputes (CSC/CA) following the destruction of its offices.

(iii) Public sector governance: In this policy area, the operation supported two sets of initiatives:

  • It sought to enforce anti-corruption laws in supporting the Anti Corruption Unit (ULCC) to improve compliance with the asset disclosure law. The government has highlighted in its governance program the need for ensuring the effectiveness of the declaration of assets law.
  • The earthquake caused significant damage to the Haiti's procurement system as the office of the procurement regulatory body (CNMP) was destroyed and the procurement units in line ministries weakened. The operation supported a number of actions to reinforce public procurement regulation and strengthen transparency in procurement practices notably restoring procurement data on CNMP website, launching the recruitment of new CNMP members, and issuing an instruction to clarify the use of accelerated procurement procedures under the State of Emergency Law.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
The total actual cost of the emergency development policy grant (DPG) was US$30.58 million greater than at appraisal (US$30). The operation was approved on August 5, 2010 and the additional co-financing of US$25 million by the Haiti Reconstruction Fund (HRF) was approved on August 10, 2010. Both operations became effective on August 23, 2010 and closed, as anticipated, on September 30, 2011. There were no extensions.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

The overall objective of the operation was and remains relevant for the following reasons:

    • During emergencies and post-disaster response, opportunities to misuse relief and aid resources typically increase, particularly when these resources are channeled through the country's budget system. It is therefore important to strengthen measures that reduce opportunities for corruption while enhancing transparency of public resource management. Such measures, including the ones supported by this operation, entail identifying bottlenecks and enhancing the integrity of government operations and budget execution systems.
    • This particular budget support operation was pragmatically focused on the short-term challenges facing the country following the earthquake. The DPG was a critical component of the World Bank's Interim Strategy Note for FY2012-13, and was aligned with the Government governance program and the National Strategy for Growth and Poverty Reduction.
    • Notwithstanding the DPG's strengths, its relevance to the larger agenda of improving government effectiveness in the recovery period needed to be clarified. The Bank should have provided a stronger rationale for prioritizing certain objectives as part of Haiti's program for improving the efficiency of public resource management with a view to promoting competitiveness and growth prospects.

IEG assesses the relevance of objectives as substantial.

b. Relevance of Design:

IEG's rating of relevance of design is based on a three-fold assessment of strengths and weaknesses:

  • Broadly, the causal linkages between the operation's objectives, its outputs, and outcomes were clear. Efforts to strengthen the internal audit function, for instance, contribute directly to re-establishing budget controls. In other areas, however, the results chain was not as clear. For instance, on the anticorruption and transparency front, the operation focused mainly on public official asset declaration. While appropriate, other complementary measures in the Government's governance program may have been equally or more effective in achieving the overarching goal of enhancing the transparency of public affairs.
  • In cases where the operation supported a longer term reform agenda (for instance, those related to public expenditure management and accountability), the design was based on relatively sound analytical underpinnings including the Public Expenditure Management and Financial Accountability Review (PEMFAR) and its follow-up action plan.
  • In spite of the loss of government employees and the destruction of infrastructure following the earthquake, policy areas designed to enhanced accountability and transparency appear to be largely a continuation of measures supported in the previous Economic Governance Operation III. Actions -- for example, those related to electricity sector transfers -- were not obviously tailored to the emergency context and the dramatic shift in fundamentals. One exception was the revision of work plan of the Internal Audit Body (IGF), which sought to adjust to the impact of the earthquake on government institutions. Policy areas and actions should have been prioritized or ranked in terms of likely impact on recovery.

IEG rates relevance of design as modest.


4. Achievement of Objectives (Efficacy) :


Overarching Objective: " strengthen and implement the legal and regulatory frameworks and systems to create the conditions for accountability and transparency" - Substantial

Overall, the project contributed to create better conditions for accountability and transparency in the context of reconstruction, notably some improvement in transparency and monitoring of financial transfers in the electricity sector, progress in reinstating budget controls by external and internal audit bodies, improved monitoring of the declaration of assets legislative for members of the executive and clarification of the use of accelerated procurement procedures under the state of emergency law.

Yet, significant challenges remain, lack of systematized control on payments to IPP and delays in the publication of data on payments as the government has not yet designated units in charge of implementing the MOUs in the electricity sector, delay in submission of the audit of the 2009-10 government accounts by the courts of accounts, limited enforcement of the asset declaration law for the members of the judiciary (50%) and the members of the legislative body (35%) and lack of information to assess whether the share of non competitive procurement has decreased to 45% as initially envisaged.

Therefore, overall, we rate the efficacy of the overarching objective as substantial. Assessments of specific objectives are provided below:

(i) Enhancing the transparency and monitoring of financial transfers in the electricity sector:

  • The first outcome required that all invoice payments by the MEF to IPP follow the rules described in joint MOU. The government has strengthened the monitoring of payments to the electricity sector through the issuance of the joint memorandum by the Ministry of Economy and Finance (MEF) and Ministry of Public Works, Transport and Communications) MTPTC, which describe the procedure to be followed for payments to IPP. The new procedure requires notably check a two levels (EDH and MTPTC) to ensure compliance with meter readings and contractual terms before the MEF can proceed with payments. The verification has saved EDH an estimated US$10 million since January 2011. This outcome indicator is met , nevertheless, the ICR noted that controls are not yet systematized and are done on an ad hoc basis due to the absence of designated units in charge of such controls and the implementation of the MOUs.
  • The operation also supported the monthly publication on the MEF website of actual payments to IPPs reconciled by the MTPTC and MEF. This outcome is partially met. Data had been published with delay of roughly three months at end 2011. This gap is due to the lack of systematized data analysis and communication between the two ministries and EDH. The ICR notes that the Bank is currently providing TA to audit the data of the monitoring mechanism.

Overall, substantial progress was made toward the objective of improving transparency of financial transfers in the electricity sector.

(ii) Reinstating budget controls
  • The ICR notes improved compliance of external audits and budget review law submission within the statutory timeframe. The MEF submitted the 2009-10 government accounts to the Supreme Audit Institution (CSC/CA) (outcome indicator met). The CSC/CA communicated to the MEF the 2008-09 audit of government accounts to the MEF, but the CSC/CA had not yet finalized the 2009-10 audit of government accounts at the time the ICR was prepared and therefore failed to meet the eight months statutory requirement (outcome partially met). Therefore the overall indicator linked to improve timely compliance of budget controls is only partially met.
  • The MEF has adopted a revised action plan for 2010-14 and a manual of internal control procedure for use by inspectors of IGF. IGF activities have complied with this new action plan and have disseminated to the competent authorities the audit reports on inventory of government and NGOs projects, audits of subsidies in the education sector and audits of public accountants activities in the MEF. The IGF had also reviewed the CSC/CA audit reports for follow-up with relevant entities. (outcome indicators met). The report notes that the IGF has started deploying inspectors in line ministries to ensure enforcement of recommendations of the internal audit reports.

Overall, substantial progress was made towards the objective of reinstating budget controls.


(iii) Advancing transparency in public affairs through enforcement of the dispositions of the declaration of asset law

  • The Anti-Corruption unit (ULCC) has certified that all members of the government (Ministers of the Cabinet) have complied with the declaration of asset law in 2011 (specific outcome indicator met).
  • However, challenges remain in this area as only 50% of the members of the Judiciary and 35% of the members of the legislative body made their asset declaration in FY2011-12. But, the operation did not consider higher-level outcome/impact indicators that, for example, measure progress in terms of overall compliance of the dispositions of the declaration of asset law including the Judiciary and legislative bodies.

Even through progress was made towards the specific outcome targeted by this operation, the efficacy of the objective is considered as modest as significant challenges still remain to lead to full enforcement of the declaration of assets law.
    (iv) Reinforcing procurement regulation and enhancing transparency in procurement practices:

    The report notes some improvement in strengthening public procurement.
    • The government has issued an instruction to Ministries and other public contracting authorities to clarify the use of accelerated procurement procedures under the State of Emergency Law and notably obtaining prior authorization from the Prime Minister. The outcome indicator which requested that the Prime Minister authorized all contracts under the emergency procedures was met.
    • As part of the prior actions, the CNMP has restored its website containing information on procurement bids and contracts, recruited new members. However, the data on procurement from line ministers were not available to check whether the share of non competitive procurement in total procurement had decreased from 56% (as of 2008) to 45%. (outcome indicator not met)

    Also, the government has adopted ten pending implementing decrees for the application of the procurement laws after the closing of the operation. For FY2010-11, five out of sixty institutions sent their annual procurement plan to the CNMP which considers their quality as low.

    Overall, modest progress was made to strengthen procurement.

    5. Efficiency (not applicable to DPLs):

    6. Outcome:

    The objectives of the emergency DPL remain relevant as the operation focuses on strengthening the legal and regulatory framework which continues to be a priority in the post emergency reconstruction period. Yet, the operation lacks strong convincing analysis to show why the selected objectives and design were the most critical in the emergency period.

    The outcome is rated moderately satisfactory based on substantial relevance of objectives, modest relevance of design and substantial achievements for two objectives. The other two objectives were rated modest for lack of information (procurement) and shortcoming in the achievement of the overarching objective of enforcement of the asset declaration law though the specific operation outcome efficacy was high (as only targeting the members of the executive).

    a. Outcome Rating: Moderately Satisfactory

    7. Rationale for Risk to Development Outcome Rating:

    The overall risk to development outcome is assessed as significant.

    The Government has demonstrated commitment to maintain macroeconomic stability and to implement the reform program supported by the DPG. But political uncertainty remains a concern as the current situation raises fears that further political gridlock hamper reconstruction activities in 2012. The ICR reports notably on stalled constitutional efforts, open conflict between the president and the prime minister and controversy with parliament after the arrest of a deputy. Institutional capacity to prepare, implement and oversee reforms remains weak. The ongoing TA to support government capacity, mentioned in the ICR as a risk mitigant, will take some time to generate some positive effects.

    a. Risk to Development Outcome Rating: Significant

    8. Assessment of Bank Performance:

    a. Quality at entry:

    The Bank team designed the emergency DPO to improve accountability and transparency in the context of the Haiti's reconstruction. The team focused on areas that were addressed in the previous operation (EGROIII) to ensure continuity and government buy-in.

    Yet, the team failed to make a good case why these issues were the critical ones in the immediate post earthquake period characterized by a drastic change in all the fundamentals of the economy. For example, as noted earlier, it would have been useful to rank the actions by largest impact to address the bottlenecks to recovery. This would have provided a more solid rationale for the selection of the objectives and the design of measures supported by the operation.

    Also, the Bank team could have better justify the use of a Development Policy Operation rather the use of another World Bank instrument such as the use of quick disbursing positive list and retroactive financing options to help Haiti with its immediate emergency financing needs. The team made the point that the government needed not only funding for emergency needs, but also support to establish a post-crisis policy and institutional framework for reconstruction. Yet, it is not clear how the operation responded to this specific need as opposed to simply continuing to align with the broader PRSP agenda and to extend the program of support provided by the previous EGRO III.

    The M&E design didn't clearly link the effect of the priority actions under the public sector governance area to the overall objective of improving transparency in public affairs.

    Quality-at-Entry Rating: Moderately Unsatisfactory

    b. Quality of supervision:

    The ICR reports that supervision visits were frequent to monitor the reform implementation status. Several aide memoire and follow-up notes were sent to MEF and the Prime Minister to emphasize the pending actions, bottlenecks and next steps. The M&E system proved useful to monitor progress.

    Quality of Supervision Rating: Satisfactory

    Overall Bank Performance Rating: Moderately Satisfactory

    9. Assessment of Borrower Performance:

    a. Government Performance:

    There was full government ownership of the program, despite the difficult policy situation which slowed down the pace of reform implementation. The MEF was the main executing agency and was responsible for the overall coordination of the program. The ICR suggests that given the difficult political/security environment and low implementation capacity, the implementation agency remained committed and did its best to carry out their activities. Yet, despite progress, there were some shortcomings in achieving all the outcomes supported by the operation to strengthen and implement the legal and regulatory frameworks. For example, the implementation of the MOU in the electricity sector was hindered by the lack of nomination of designated units to ensure controls were systematized and data payments were monthly reconciled and published on the website.

    Government Performance Rating: Moderately Satisfactory

    b. Implementing Agency Performance:

    Implementing Agency Performance Rating: Not Applicable

    Overall Borrower Performance Rating: Moderately Satisfactory

    10. M&E Design, Implementation, & Utilization:

    a. M&E Design:

    The M&E design didn't clearly link the effect of the priority actions under the public sector governance area to the overall objective of improving transparency in public affairs. The specific objective was to enforce the law on declaration of assets. The document doesn't provide a strong rationale on why this was the most critical measure to focus on during this emergency period after the earthquake. The team mentions that this objective is aligned with the governance program presented in New York in March 2010 but didn't provide more specifics on other measures of that program that also foster transparency and could have received support from the operation. The selected outcome indicator only focused on full compliance for executive members which was already at 87% in May 2010 but didn't retain any progress on compliance for members of the judiciary and legislature which compliance rate was much lower respectively 50% and 35%.
    The program document was not very specifics on how the M&E framework would be implemented including the collection of relevant data.

    b. M&E Implementation:

    Baseline data were not collected for the procurement outcome indicators. At the operation inception, data on procurement from line ministries were not available as the earthquake damaged the procurement data management system at the CNMP and line ministries. The timeframe for the procurement outcomes proved to be too optimistic.

    The Bank team closely monitored progress with implementation of the reform program. The supervision missions used the monitoring indicators to assess progress while at the same time the overall status of the government program was monitored under the review of the joint matrix of Haiti's partners to determine whether country conditions and the specific policy actions of the operation were met.

    a. M&E Utilization:

    Implementation progress was tracked through the M&E system and based on the findings, the team follow-up with the government through regular aide memoire to report on pending actions and problems that needed to be addressed.

    M&E Quality Rating: Substantial

    11. Other Issues:

    a. Safeguards:
    The operation triggered none of the Bank's safeguards.

    b. Fiduciary Compliance:

    The operation was carried out in compliance with the Bank's fiduciary standards. The Program Document notes that in February 2010 an exception was granted by the World Bank Management as the audit reports of the financial statement of 11 projects became overdue after January 31, 2010. At the time the PD was completed, the audit of three reports had been submitted, the requirements for two waived on the ground of limited activity and the audit of the remaining six projects underway. The IMF was also preparing an audit of the Central Bank of Haiti. The ICR doesn't report on any fiduciary issues related to financial management including the conclusions of the remaining audits and the latest IMF safeguard assessment.

    c. Unintended Impacts (positive or negative):
    NA

    d. Other:
    NA



    12. Ratings:

    ICR
    IEG Review
    Reason for Disagreement/Comments
    Outcome:
    Moderately Satisfactory
    Moderately Satisfactory
     
    Risk to Development Outcome:
    Moderate
    Significant
    Significant risks remain notably the fragile political and security environment. Though macroeconomic risks with the operation have been abated, risks related to weak capacity remain challenging and may jeopardize the expected impact of the DPG. The ongoing TA to support government capacity, mentioned in the ICR as a mitigating strategy, will take some time to generate some positive effects.  
    Bank Performance:
    Satisfactory
    Moderately Satisfactory
    The rating on quality at entry brings the overall rating down. 
    Borrower Performance:
    Moderately Satisfactory
    Moderately Satisfactory
     
    Quality of ICR:
     
    Satisfactory
     
    NOTES:
    - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
    - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

    13. Lessons:

    IEG mainly supports lessons discussed in the ICR. There are two important lessons:
    • Simple and straightforward policy based program to restore institutional functions as fast as possible in a country experiencing major natural disasters (lesson noted in the ICR).
    • Setting priorities to ensure strong government ownership and donor support along the critical path to improved government effectiveness in the immediate recovery period (lesson formulated by IEG).

    14. Assessment Recommended?

    Yes
    Why?

    Some policy areas covered under this operation were addressed in the previous EGROIII. A clustered Project Performance Assessment Report (PPAR) that covers both operations can shed light on incremental progress made, as well as lessons in adapting policy based operations to emergencies.

    15. Comments on Quality of ICR:


    The ICR provides adequate analysis of the context and implementation of the reform program. It could have made a better case for justifying why the DPG objectives and design were the most relevant in the immediate recovery period after the earthquake as well as in describing the design of the M&E system.

    a. Quality of ICR Rating: Satisfactory

    (ICRR-Rev6DPL-Jun-2011)
    © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions