Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Institutional Reform Development Policy Grant


  
1. Project Data:   
ICR Review Date Posted:
11/13/2012   
Country:
Yemen
Is this review for a Programmatic Series?
 No
First Project ID:
P101453
Appraisal
Actual
Project Name:
Institutional Reform Development Policy Grant
Project Costs(US $M)
 50.93  26.99
L/C Number:
CH336
Loan/Credit (US $M)
 50.93  26.99
Sector Board:
Cofinancing (US $M)
 0  0
Cofinanciers:
None
Board Approval Date
  12/06/2007
 
 
Closing Date
06/30/2010 09/30/2011
Sector(s):
Central government administration (50%), General industry and trade sector (50%)
Theme(s):
Administrative and civil service reform (23% - P) Regulation and competition policy (22% - P) Personal and property rights (22% - P) Public expenditure financial management and procurement (22% - P) Tax policy and administration (11% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
S. Ramachandran
Robert Mark Lacey Navin Girishankar IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The objectives of the Institutional Reform Development Policy Grant (IRDPG), according to the Program Document (page 19), were (i) increasing non-oil growth (with special attention to recommendations included from the recent investment climate assessment); and (ii) strengthening governance. These objectives are consistent with the overall goals of the Government's National Reform Agenda as stated in the Letter of Development Policy (page 2) "to improve Yemen's investment climate and strengthen its governance and democratic institutions."

b. If this is a single DPL operation (not part of a series), were the project objectives/key associated outcome targets revised during implementation?
No

c. Policy Areas:

The main policy areas the project sought to address were (i) administrative and civil service reforms, (ii) personal and property rights (specifically clarifying the law applicable to the private ownership of land), (iii) public expenditure, financial management and procurement, (iv) regulation and competition policy, and (v) tax policy and administration (particularly transparency in oil revenues).

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Originally conceived as an IDA adjustment credit (even at negotiations in May 2007), the IRDPG became a two tranche grant before it was approved by the Board in December 2007 because grant funding became available. The first tranche of $26.99 million (half the total SDR equivalent) was released on July 17, 2008 (ICR page 9, although the chart on page 16 has June 2008),

The second tranche release required the implementation of approved reform measures within 12 months of signing (i.e. by December 2008). The grant was extended by three months to September 2011 to allow the Government more time for meeting second tranche conditions (ICR, page 9). At the end of that month, with no assurance that the macro-economic situation was or would soon become sound, the grant was closed without the second tranche having been disbursed.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Substantial.
The project's broadly stated objectives (non-oil growth and better governance) were relevant and consistent with the first of the four pillars of the Bank's Country Assistance Strategy for 2006-09, namely "diversifying growth". Yemen's oil reserves were being depleted, and governance was thought to be an impediment to realizing faster non-oil growth. The CAS anticipated that strengthening of public administration accompanied by an enhancement of the investment climate -- both key project goals -- would contribute to reducing fiscal risk, by helping to contain the overall wage bill and strengthening service delivery, and by contributing to non-oil growth, and hence medium- and long-term revenue. In addition, the experience of implementing the Grant, combined with planned economic and sector work, was expected to help the Bank to assess the readiness of Yemen for subsequent programmatic lending,

b. Relevance of Design:

Modest
The design of the project suffered from the absence of a clear causal chain connecting the measures being supported to the outputs, outcomes and ultimately the objectives. Non-oil growth was already around 5.3% in 2004 and 2005 (Project Document, Table 1), and increasing it would require a range of measures to raise investments (including in human capital) and savings that are constrained by economic fundamentals (e.g greater savings requires lower consumption). The project documents do not discuss such constraints. Removing hurdles to business identified through investment surveys may be of relatively insignificant economic importance compared to them.

The project supported specific changes grouped in five clusters ("policy areas" under 2c), and while all of them are desirable, they are (i) not of equal economic importance, and (ii) may not be either necessary or sufficient for non-oil growth. Some measures have immediate benefits: improving procurement practices would reduce misappropriated and wasted resources, and greater transparency of the oil company's finances would improve governance (revenues from oil were about 29% of GDP in 2006, and enhanced oil company transparency is one of the requirements for compliance with the Extractive Industries Transparency Initiative – see Section 4 below). Other measures are not so clearly linked to the anticipated outcomes: greater transparency of government revenues and spending may not improve governance significantly unless it were to be accompanied by informed public choice in the political process. Although civil service reform could improve the fiscal situation in the long term, the ICR acknowledges that the design of the civil service component "was too complex and technically flawed" (page 22). Moreover, the design of the reforms did not take account of the lessons the Bank learnt from its experience in other countries, including the importance of broad public support for the reform process. The law that secures private ownership of land is important and necessary, but clarifying the de jure ownership of land may not result in much gain if courts do not function efficiently and enjoy public confidence.

Even if all measures supported by the grant were to have been undertaken and all expected outputs attained, it is unclear whether they would have been sufficient, in and of themselves, to achieve a significant increase in non-oil growth.


4. Achievement of Objectives (Efficacy) :


A. Increasing non-oil growth. Negligible

Outputs
    • A new land registration law encompassing the recommendations of the Land Policy Task Force was submitted to Parliament. However, it had not been approved or enacted by project closure.
    • As a consequence, the General Authority for Land Survey and Urban Planning still lacks the legal basis to secure and enforce property rights.
    • The project supported the reduction of the marginal corporate income tax rate from 35 to 20 percent and a clarification and simplification of several provisions. However, it is unclear how many firms this would affect and what would be the effect on government revenue. Other factors affecting the impact of the reductions -- interaction with other taxes in the economy, the reliability of company accounts and efficiency of administration – receive little attention in the project documents, and there are no data on their economic effects.

Outcomes
    • The ICR contains no data or discussion on the extent to which non-oil growth may have increased or decreased, nor of the impact of project-supported measures on such trends.
    • The delays in implementation of a number of second tranche actions meant that the 2010 enterprise survey, which was to afford some basis for evaluating results, was completed before the relevant reforms were in place or could have been expected to have had a substantive impact on the ground.

B. Strengthening governance. Modest.

Outputs
    • Administrative and civil service reforms were carried out with a view to achieving a leaner and better functioning civil service, and also to eliminate “ghost workers” and “double dippers” (those on multiple payrolls). However, the ICR acknowledges (p. 26) that "the intended objective of a leaner civil service was not achieved." The main instrument chosen was bio-metric identity cards for all public employees. But coverage was less than 100%. The Government, in the Borrower’s ICR states (ICR, page 39) that "98% of the civil service has been integrated in the bio-metrics database as well as [the] military service." The project team subsequently informed IEG that a World Bank expert worked with the Civil Service Ministry on the bio-metrics data base. However, the ICR states (page 26) that "while the bio-metric system grew to cover 95% of all public employees, it is not known if the remaining uncovered group was biased towards double dippers and ghost workers." This would suggest that the figure of 98% coverage has not been verified by Bank staff.
    • An organization to oversee public tenders (Higher Authority on Tender Control), supported by a Procurement Management Information System, was created along with a website to make information easily accessible. However, the website mentioned in the ICR has not been updated for over a year. “Since the permanent Procurement Management Information System was in pilot stage when the conflict broke out, it is unlikely to have progressed much” (ICR, page 25). It was planned that foreign consultants would visit the country to make the System fully operative, but these visits were not possible due to the security situation.
    • Yemen acceded to the Extractive Industries Transparency Initiative in March, 2011. The country was suspended in June, 2011, with the EITI Board citing concerns that it was "not satisfied that the full and active participation of civil society and other actors in EITI implementation could be maintained" (ICR, page 26). The EITI website (September 2012) reports that the suspension was lifted in June 2012, but there has been no EITI report since the single one issued in 2010 covering 2005 to 2007.

Outcomes
    • The ICR (page 23) acknowledges that, due to the civil conflict, “[the ICR is] prepared in ignorance of the detailed status of the IRDPG components and their impact on the ground. In several cases, given the preoccupation of officials with a crisis, it is safe to assume that no further progress has been achieved since early 2011. In other cases, there may have been backsliding of which we are as yet unaware. In no case is there a firm empirical or analytical basis to evaluate impact.”
    • There is no evidence that the civil service reform has led to a leaner, more efficient or more transparent administration. There are, however, indications that the reform process did not enjoy public support because of lack of access to, and knowledge of, what the process meant to affected parties. In April 2009, the Bank’s Inspection Panel received a complaint from the Yemen Observatory on Human Rights concerning the fact that the Government’s obligations under the Grant Agreement had not been made publicly available and not been translated into Arabic. After a visit in June 2009, the Bank's Inspection Panel issued its report in September 2009 finding that no safeguards were violated but suggested that public documents be made available in Arabic.
    • The ICR states (page 27) that “the EITI initiative created a tri-partite governance structure -- the EITI Council -- which gave a stronger role to civil society (and to the private sector) than was traditionally experienced in Yemeni policy contexts. It created the procedures and mechanisms for continuous monitoring of extractive industry payments and revenues.” However, no evidence is provided concerning the functioning of this mechanism.
    • There are no indications provided from independent sources (such as Transparency International or the Bank’s own governance indicators) of any changes in the perception of the quality of governance in Yemen.

5. Efficiency (not applicable to DPLs):

6. Outcome:

Although the objectives of higher non-oil growth and improved governance are substantially relevant, it is unclear whether the policy measures supported by the Grant would be necessary or sufficient to attain those goals. Design relevance is, therefore, rated modest. With regard to the first objective – higher non-oil growth -- many specific outputs and outcomes that were sought in each of the five policy areas were not attained. No evidence is provided that non-oil growth has increased, and, given the partial project results, higher growth would be difficult to attribute to the project even if it had occurred. The efficacy of this objective is, therefore, rated negligible. With regard to governance, there were some positive results, such as increased understanding of civil service issues and Yemen’s adhesion to the EITI. However, the ICR acknowledges the lack of evidence concerning the impact of project activities on the ground. The efficacy of this objective is rated modest. Overall outcome is assessed as unsatisfactory.

a. Outcome Rating: Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

The Project Document lists policy reversal, loss of reform momentum, weak institutional capacity and fiscal and macroeconomic effects of exogenous shocks that include regional conflicts, as significant project risks. The nature of the reforms being attempted (e.g. of the civil service, tax and public administration) required conditions that were unlikely to hold. The deterioration in the security situation since then has impeded staff travel and even the modest gains under the project are unlikely to be sustained because government counterparts change more rapidly, and their knowledge of the issues disappears with their departure.


a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:

The project's quality at entry had two major shortcomings. First, a complex civil service reform would be attempted without taking account of the lessons learnt from the Bank's experience elsewhere of the importance of broad public support that is needed to sustain the effort

Second, the intention of ensuring the security of private property with a satisfactory law was not fulfilled.The initial project document sought "a first tranche requiring legislative achievements as prior actions, and the second tranche focused on the implementation of approved reform measures within 12 months of signing to assure that supported measures are implemented." (paragraph 40 of the Project Document). However, the project was sent to the Board before the law was approved. When Parliament refused to approve the law (reportedly over disputes unrelated to the law's content, since a task force drafted it with adequate consultations), the first tranche could not be witheld. Although a law's passage could not be a tranche release condition, it could have been a condition of Board presentation or Grant effectiveness.

More thorough scrutiny by macro-economists during preparation may have questioned how non-oil growth could result from the measures being supported. Both the Project Document and the ICR begin with a sentence that "Yemen's extreme poverty demands an urgent response," implying that a quick response (rather than sustained development) would alleviate poverty.

These major shortcomings in project design and structure were not corrected during the internal review process.

Quality-at-Entry Rating: Unsatisfactory

b. Quality of supervision:

Page 30 of the ICR reports how, despite civil conflict and the dangers of mission travel, staff supervised the project adequately using local staff. There was a good working relationships with government counterparts. The difficulties in implementing the program supported by the grant (notably the civil service reforms) , reflected design shortcomings rather than insufficient supervision. More effort could have been made in systematically monitoring and evaluating progress towards the project’s stated objectives – higher non-oil growth and better governance – but no resources were allocated for this and supervision missions were not fully equipped for it.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

While some government officials made efforts to implement the reforms supported by the project, they did not always have the cooperation of relevant officials in other Ministries. Civil service reforms in particular were undermined by inadequate and unreliable numbers, and there was less than full commitment on the part of some government counterparts to achieving greater clarity. The Government’s confidence that it could ensure the passage of the land law proved unfounded.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:

The ICR (p.31) reports that implementing agency performance varied strongly between agencies. The
Tax Authority and Ministry of Finance were “highly cooperative, active and politically courageous at critical stages,” although the tax component actions were significantly delayed beyond their original timeframe. The EITI component was subject to substantial delays due to lack of consensus within the Yemen EITI council, although ultimately the trigger of adhesion to the Initiative was achieved. The Public Procurement component relied on the creation and capacity building of a new government institution, which took longer than originally predicted and could not be completed. The civil service component suffered from “a failure of the Government to abide by its own established policies and laws” (ICR, page 31), as well as a persistent lack of accurate monitoring and reporting. Overall coordination of the reform effort was the responsibility of the Ministry of Planning and International Cooperation. While this Ministry did its best to exercise its project-related responsibilities, it did not have line authority over other government entities, from which cooperation was variable.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The Program Document makes no reference to monitoring and evaluation of outcomes, and while there was considerable effort to monitor progress towards meeting the conditions of second tranche approval, there was no attempt to monitor the measurable non-oil growth objective.

The Project Document's Table 3 is a long list of Good Governance Measures and Achievements. Even leaving aside the merits of the indicators and the lack of prioritization, there was no attempt to measure progress subsequently.

b. M&E Implementation:

Monitoring and evaluation were largely limited to following progress in meeting tranche release conditions. There was a considerable effort to gather data on civil service staffing, but even this did not benefit from systematic and accurate monitoring and reporting. It is unknown whether information has been kept up-to-date.

a. M&E Utilization:
No information is provided on utilization of the project's M&E system or of the data collected by it.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
NA

b. Fiduciary Compliance:
NA

c. Unintended Impacts (positive or negative):
NA

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Unsatisfactory
Unsatisfactory
Although the relevance of the operation's objectives was substantial, that of design was modest given the absence of a clear causal chain between the activities supported by the Grant and attainment of the objectives. Efficacy of the first objective -- higher non-oil growth -- is assessed as negligible, since no attempt was made to gauge its achievement, while that of the second -- enhanced governance -- is rated modest based on the very limited evidence available. 
Risk to Development Outcome:
High
High
 
Bank Performance:
Moderately Satisfactory
Unsatisfactory
Quality at entry presented major shortcomings: there was no clear causal chain between the reform measures supported and the operation's objectives; the law needed to secure private land was not made a condition of Board presentation or Grant effectiveness; and a complex civil service reform was attempted without taking account of the lessons of Bank experience elsewhere, especially regarding the need for broad public support. 
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Unsatisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The main lessons from the experience of preparing and implementing this operation are:

1. A tight causal chain linking the measures being supported to the objectives that are sought is an important dimension of good preparation. Without this, much effort could be expended on measures of little consequence. In this case, addressing deep rooted poverty or increasing non-oil growth require a wel- considered strategy and a sharp focus in project design.

2. Better economic analysis using available data would have focused the project and set priorities. The team could have benefited from the participation of a macro-economist who would have analyzed the extent to which the measures being supported could lead to higher non-oil growth.

3. "Stroke of the pen" measures (e.g. passing a law, reducing a tariff) and those that require persistent effort should be distinguished. Civil service reform in particular requires consistently good budgetary oversight and continued broad public support, and it should have been clear at the outset that neither was present.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR is well written and narrates project-related developments clearly and concisely. However, a major weakness of the ICR is the lack of an attempt to assess achievement of the operation’s objectives as written, especially with regard to higher non-oil growth. The ICR states that data were unavailable because of the civil conflict in 2010, with related restrictions on travel and other disruptions. However, International Monetary Fund documents (Article IV consultation in January 2010, an Extended Credit Facility approved in August 2010 and a $93 million Emergency Loan in April 2012) contain data that could have been used directly, and Fund staff would have been able to provide further information had they been approached.

a. Quality of ICR Rating: Unsatisfactory

(ICRR-Rev6DPL-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions