Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Ry-second Vocational Training Project


  
1. Project Data:   
ICR Review Date Posted:
05/28/2013   
Country:
Yemen
PROJ ID:
P086308
Appraisal
Actual
Project Name:
Ry-second Vocational Training Project
Project Costs(US $M)
 15.00  0.37
L/C Number:
C4279
Loan/Credit (US $M)
 15.00  0.37
Sector Board:
Education
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  03/27/2007
 
 
Closing Date
05/31/2013 08/03/2012
Sector(s):
Vocational training (68%), Central government administration (32%)
Theme(s):
Education for the knowledge economy (50% - P) Improving labor markets (25% - S) Micro Small and Medium Enterprise support (25% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Susan Ann Caceres
Judyth L. Twigg Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Project Appraisal Document (pp. v and 5), the objectives were to provide the public training system with improved mechanisms for responsiveness to employment needs, and to enhance the capacity and effectiveness of the Skills Development Fund in articulating and financing the training needs of enterprises. The same objectives were noted in the Financing Agreement (p. 5).

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

There were three components (actual costs are not provided below, since this operation was cancelled):

1. Monitoring and Evaluation, and Project Management (appraisal, US$ 3.01 million) contained two subcomponents:

    • Monitoring and Evaluation System was to help address two existing constraints to the viable expansion of vocational and technical services: (1) weakness of the information base for decision-making on the costs and outcomes of training and employment needs; and (2) lack of appropriate incentives to promote demand-responsiveness and cost effectiveness. This was to be done by building relevant capacity of a team to carry out monitoring and evaluation in the Ministry of Technical Education and Vocational Training, as well as development of a management and information system, tracer studies, and employer surveys.
    • Project Management was to ensure effective administration and coordination of project activities, including reporting requirements. It was to establish a Project Management Unit staffed by national and international short-term consultants.

2. Design and Delivery of Training Programs (appraisal, US$ 6.13 million) had three subcomponents:
    • Technician Level Programs were to pilot the development and delivery of new international-standard programs in four existing training institutes to meet identified needs in the sectors of construction and food processing. It was to finance a twinning program to provide technical services for project management and program development, training, and fellowships for instructional staff. It was to construct a workshop at the Hada Technical Institute, provide minor rehabilitation of other facilities, and purchase instructional equipment and supplies.
    • In-Service Training was to pilot approaches for improving the capacity of public sector training providers to respond more effectively to employer-defined in-service training needs in the target economic sectors of construction, food processing, fisheries, and seafood processing. Skills and Employment Liaison Units were to be established in the four training institutes to undertake outreach activities with local enterprises to identify training needs and develop market services to meet these needs.
    • Instructor Development was to pilot approaches in four existing training institutes to strengthen the occupational skills and competencies of a cadre of 48 Master Instructors in the fields of civil works, commerce, accounting/business administration, electrical and electronic maintenance, and mechanical maintenance. This was to fund training costs for work placement, a fellowship program, equipment, and employer participation in program design and delivery.

3. Reform of the Skills Development Fund (appraisal, US$ 1.85 million) was to finance a long-term advisor, short-term technical assistance, workshops and training, and equipment to support restructuring the Skills Development Fund. It was to build the capacity of the staff of the Skills Development Fund and establish a management information system.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Costs: Project costs were planned to be US$ 15 million, with an estimated US$ 9 million in foreign expenditures and US$ 6 million in local costs (NCO, p. 11). Before cancellation, US$ 0.37 million was disbursed.

Financing: The project was to be financed by an IDA credit of US$15 million.

Borrower Contribution: No contributions were planned or made.

Dates: The credit was approved on March 27, 2007, and the legal documents were signed on June 7, 2007. However, the effectiveness conditions took time to satisfy (Subsidiary Agreement executed on behalf of the Recipient and Project Implementing Entity; Project Management Unit established by the Recipient with acceptable organization, staffing, and terms of reference; and passing of the Skills Development Fund law). As a result, the credit effectiveness date was extended three times from the initial date of October 5, 2007 to October 6, 2009. The NCO does not provide reasons for this delay, which permitted only 3 years and 7 months to implement a project originally planned for six years. Given the delay in effectiveness, the project was in jeopardy of not achieving its development objectives, since design was based on development and delivery of a new three-year technician-level program that would have required two years to develop (through technical assistance selection, contracting, mobilization, program development, equipment procurement, and staff training) before students could then enroll.

Two requests for proposals related to Component 2 were issued in 2010 but yielded no proposals, and all short-listed firms cited "factors external to the project" (declining security conditions in the country) as the reasons for non-submission. Consensus between the Bank and the Implementing Agencies was reached in 2010 to restructure the project to reduce its scale, simplify the components, and reduce the credit to US$ 9 million, but it was not until May 11, 2011 that the Government forwarded a request to the Bank to restructure the project. There was a suspension of disbursements in July 2011 due to unrest in the country, as Bank staff had been relocated to Amman in March 2011. The operation was cancelled on September 3, 2012, two years and 10 months after effectiveness. Ultimately the project was cancelled because: (1) most project activities could not be completed and the objectives achieved before the planned closing date, (2) difficulties faced by the project were likely to continue due to capacity and security issues, (3) there was a lack of local expertise to undertake project consultancies, and (4) there were difficulties in hiring qualified firms or international consultants willing to travel to Yemen.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Objectives: Substantial

The project's objectives were substantially relevant to the Bank’s current Country Assistance Strategy (2010-2013), which stresses access, quality, and efficient use of public resources across all levels of education, including vocational. At the time of appraisal, the main issues in the vocational and technical education sub-sector were the need to: (1) diversify programming to respond to a wider range of social and sectoral partners; (2) develop linkages with the labor market at a number of levels to ensure that all activities of the system were driven by employer and employment demands; (3) review the governance management and organizational structure of the Ministry of Technical Education and Vocational Training to reflect the requirements of an employment-driven marketplace; (4) reorient and re-equip the network of Technical Education and Vocational Training institutions to meet a wider range of employer and community skill development needs; and (5) diversify the financing of Technical Education and Vocational Training in order to share costs more equitably among beneficiaries. The project was also directly aligned with the Government's priorities as outlined in its Technical Education and Vocational Training Strategic Development Plan and Development Plan for Poverty, which focuses on strengthening the links between technical education and the private sector.

b. Relevance of Design:

Design: Modest

Design included activities intended to build capacity, develop training programs, pilot mechanisms for employer participation in the design and delivery of new pre-service and in-service training, and revise the Skills Development Fund, all of which were logically related to the objectives. The design was too complex for the capacity and conditions (NCO, p. 15). The project underwent a Quality Enhancement Review, raising several other design weaknesses that were not addressed. These issues included: relevance of the project to skill development needs, current labor market demand for graduates, incentives to ensure employer collaboration, capacity of the Skills Development Fund to undertake a reorientation of its service delivery model, and the need to improve the project outcome indicators. Outcome indicators did not effectively demonstrate attainment of the objective (see Section 10a).


4. Achievement of Objectives (Efficacy) :

N/A

5. Efficiency:

N/A

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
%
%
ICR estimate:
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

N/A

a. Outcome Rating: Not Applicable

7. Rationale for Risk to Development Outcome Rating:

N/A

a. Risk to Development Outcome Rating: Not Applicable

8. Assessment of Bank Performance:

a. Quality at entry:

The Bank was familiar with the education sector through its previous projects and analytic work related to other sub-sectors within education. Preparation was supported by a Japan Policy and Human Resources Development grant that provided resources for further analysis of institutional capacity and engagement of a wide range of stakeholders, including the employer community and representatives of the nationally identified economic sectors of fisheries, construction, and food processing. Resources from the UK Department for International Development were used to develop operational procedures for the Skills Development Fund.

The Bank team worked closely with the Ministry of Technical Education and Vocational Training to formulate policy objectives to guide the project and facilitate a comprehensive analysis of the Skills Development Fund (legal, regulatory, institutional, management, administrative, and operational aspects), as well as dialogue and cooperation between the Ministry and the Fund. However, differing approaches resulted in a lack of understanding and commitment related to employer-community partnership (NCO, p. 17). Risks and mitigation measures were appropriately identified; however, preparation over-estimated the readiness and capacity of the implementing entities.

When the credit effectiveness conditions were not satisfied, the Bank tried several interventions, including Bank management convening a high-level meeting. During the delay of credit effectiveness, the Bank team, Ministry of Technical Education and Vocational Training, and the Skills Development Fund maintained close contact.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

The Bank team provided extensive support to the Ministry of Technical Education and Vocational Training and Project Management Unit during the period of supervision, but there was a high turnover of Bank TTLs, impacting supervision (NCO, p. 15). The Bank team provided regular supervision missions (until the staff were relocated) and provided technical and administrative support to the Project Management Unit and the Ministry. The Bank team provided guidance on preparation of Terms of Reference, identified potential consultants, and assisted with contracting procedures. The Bank team expressed concerns when issues emerged, making adjustments to the implementation approach to the selection of the management information system under the first component. The Bank also drew up action plans to address weaknesses in procurement capacity within the Project Management Unit. The Project Management Unit Director was dismissed, which was a joint responsibility of both the Bank and the Ministry. Given the lack of implementation progress, limited remaining time, and slow rate of disbursements, the Bank team reached consensus with the Ministry of Technical Education and Vocational Training and Skills Development Fund to restructure the project to reduce its scale and simplify design. The Bank team prepared the restructuring package, but it was not processed for approval, since at the same time the country's security conditions had deteriorated and Bank staff were relocated to Amman. The Bank team continued to provide regular support to the Ministry and Project Management Unit through email and audio conferences, but there was a slow-down in the Bank's support preceding the suspension (NCO, p. 16). The Bank suspended disbursements for the project on July 28, 2011, since it could not fulfill its fiduciary responsibilities. Agreement was reached in May 2012 to cancel the project.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

During preparation, there was constructive dialogue with the Bank at the vice ministerial level, where there was a clear understanding of the sector and strong leadership and decision-making skills. There was need for more intensive intervention during the period of delay in credit effectiveness by the executive management level of the Skills Development Fund, Ministry of Technical Education and Vocational Training, and Ministry of Planning and International Cooperation, since the lost time placed the whole project in jeopardy. While consensus was reached between the Bank and Ministry of Technical Education and Vocational Training and the Skills Development Fund to restructure the project in 2010, the Government did not submit the formal restructuring request until May 11, 2011.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:

Two parallel implementation arrangements were established: the Ministry of Technical Education and Vocational Training, with the support of a Project Management Unit, and the Board of Directors of the Skills Development Fund, as well as oversight by a national Steering Committee. During the delay of credit effectiveness, a series of activities was undertaken to enhance implementation readiness, including staffing the offices of the Director, Technical Advisor, Procurement Officer, and Financial Officer of the Project Management Unit, review of physical facilities in pilot institutes, preparation of terms of reference, survey of pilot institutes, and dialogue with employers in target sectors. The Skills Development Fund drafted bylaws and a transition plan.

There were several weaknesses in performance. First, there were issues with staff within the implementing entities. The Project Management Unit was never fully staffed with individuals containing the required knowledge, skills, and experience. The initial appointment of the Director of the Unit was delayed for one year. Staff in the Ministry changed, which impacted decision-making and resolution of issues, and momentum was lost as the incoming team took time to learn about the project. There was only one employee who was charged with implementation of the revised Skills Development Fund law, which was contrary to the organization model in the operational manual. The Skills Development Fund established a new executive Board and developed terms of reference for a short-term advisor, rather than a long-term advisor as planned. No new staff were appointed to the Skills Development Fund, except for a finance officer. The Project Management Unit Director was dismissed, which resulted in the loss of project history; the NCO does not provide reasons for this dismissal. A temporary Acting Director was then installed, which was an inadequate arrangement and impacted project restructuring.

Second, there were numerous delays, including in the purchase of a financial management system. There was slow implementation progress that related to the lack of firms submitting proposals to implement Component 2, as well as adjustments to the approach to implement Component 1, which were made by a short-term policy advisor in collaboration with the Ministry of Technical Education and Vocational Training.

Third, there were issues with engagement and capacity. Procurement support remained weak, despite action plans drawn up by the Bank after each supervision mission. The Board of Directors for the Skills Development Fund had limited involvement in project implementation. Few members from the employer community attended Board meetings. The operation manual was not endorsed and there was no initiative to promote its procedures.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The Results Framework did not adequately measure either of the project's objectives. Instead, the Results Framework contained measures of outputs of the components (e.g. new program established, diversity of in-service training, establishment of management information system, revised organizational structure), rather than outcomes to demonstrate improvements in the training system and Skills Development Fund. The Project Appraisal Document (pp. 30-32) specified the data collection timeline and assigned responsibility for collecting the data. Baseline and targets were provided, but the utility of this information was low, given how they were stated and the limited increase anticipated.

b. M&E Implementation:

a. M&E Utilization:

M&E Quality Rating:

11. Other Issues:

a. Safeguards:
N/A

b. Fiduciary Compliance:
N/A

c. Unintended Impacts (positive or negative):
N/A

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Not Rated
Not Applicable
 
Risk to Development Outcome:
Non-evaluable
Not Applicable
 
Bank Performance:
Moderately Unsatisfactory
Moderately Satisfactory
The shortcomings presented in the NCO mainly related to design and its complexity (which is rated under Relevance of Design). As well, the shortcomings in performance during preparation and supervision were minor. Given the delay in the Government's formal restructuring letter and the deteriorating country conditions, the timing of the actions of the Bank were reasonable. 
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The NCO (pp. 17-21) contains several lessons, which IEG summarizes as follows:
    • Capacity in Yemeni institutions to implement change is low. In low-capacity contexts, design should incorporate modest objectives and be simple in construct. Component two was based on best international practices for technical education and vocational training with employer partnerships, identification of occupational competencies and standards, development of learning assessment tools, and provision of learning resources and equipment. The delivery contained a twinning arrangement with an international consortium supporting both program development and implementation, and providing management support while counterpart institutional capacity would be built through working groups drawn from the respective directorates of the Ministry, chaired by the deputy Minister. The partnership-based approach proved to be beyond the capacity and experience of the Ministry of Technical Education and Vocational Training. Both country and sector context should be understood and factored in when developing project components, activities, and implementation arrangements.
    • Investment in leadership is critical. Bank-funded development projects are inherently multi-dimensional in nature and call for mature leadership throughout their implementation. The current pool of project management and leadership skills in Yemen, as reflected by advisements for the Project Management Unit staff, local technical assistance providers, and local consultants, is neither deep nor broad. This suggests that the capacity of all appointees needs to be carefully assessed and supported by relevant professional development experiences that can serve as both an incentive and a pathway to professional growth, thereby enhancing project performance.
    • Local language skills are an essential requirement for project ownership and comprehension. Many of the policy issues, design concepts, implementation details, and operational modalities for the project were discussed with the partners primarily in the English language. The majority of documents designed to capture these discussions were similarly prepared in the English language. This being the case, it was frequently evident that those responsible for implementing the project were not fully aware of the purpose and expected outcomes of the responsibilities to which they were assigned. This problem was compounded by an internal culture in both the Ministry and the Skills Development Fund that discouraged open discussion and information sharing. While this is a lesson that applies to all languages and cultures, it is of particular importance in Yemen.
    • Gaining access to employment opportunities in the Gulf region requires a business approach rather than an administrative, institutional, or bureaucratic approach. Adoption of this principle suggests that the management of training and employment placement might be led by a private sector entity, while management of regulatory and work permit issues could be led by the relevant Government Ministry.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The Note on Cancelled Operation is thorough, candid, and well written. It discusses the main events leading to cancellation and the steps taken to resolve the problem. It analyzes the events and presents a rich discussion of the causes of the problem and the actions of the parties involved. The Note is filled with insightful analyses and lessons, and its content could be very useful to promote learning for operational staff in Yemen, the MENA region, and beyond.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions