Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Dominican Republic Youth Development Project


  
1. Project Data:   
ICR Review Date Posted:
04/29/2014   
Country:
Dominican Republic
PROJ ID:
P096605
Appraisal
Actual
Project Name:
Dominican Republic Youth Development Project
Project Costs(US $M)
 49.9  33.2
L/C Number:
L7371
Loan/Credit (US $M)
 25.0  23.0
Sector Board:
Social Protection
Cofinancing (US $M)
 10.0  9.4
Cofinanciers:
IADB
Board Approval Date
  03/23/2006
 
 
Closing Date
06/30/2011 06/30/2013
Sector(s):
Vocational training (66%), Adult literacy/non-formal education (21%), Central government administration (13%)
Theme(s):
Improving labor markets (50% - P) Education for all (25% - S) Vulnerability assessment and monitoring (25% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Hjalte S. A. Sederlof
Judyth L. Twigg Lourdes N. Pagaran IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Loan Agreement, the Project Development Objective (PDO) was to improve the employability of the Borrower’s poor at-risk youth population through: (i) building their work experience and life skills; and (ii) expanding the second-chance education programs to complete their formal education.

    The PDO in the Project Appraisal Document (PAD) was substantively the same.

    The project was restructured twice. During the first restructuring, in September 2010, the PDO was revised to read as follows: The Project Development Objective is to support the Borrower in improving the employability of (a) poor at-risk youth by building their work experience and life skills and expanding second-chance education programs to complete their formal education, and (b) poor low-skilled long-term unemployed people between the ages of 18 and 65.

    The revised PDO retained the original PDO and introduced a new objective. Project outcomes will be assessed against the original and revised project objectives.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 09/15/2010

c. Components:

Original project components:

Component 1: Consolidate and expand the Youth and Employment Program (PJE) to provide increased access to a first work experience for youth at risk (estimated cost at appraisal US$ 22.08 million; actual cost US$ 15 million).

This component was to expand the country’s labor market insertion program for disadvantaged youth (PJE). Independent training institutes were to be contracted to provide vocational training, internships, and life skills training needed to gain first time experience in a job. The program was also to include pilot training courses and internships to promote entrepreneurship and self-employment. Approximately 32,000 youth of age 16-29 were to benefit from the expanded program over a period of four years.

Component 2: Build capacity in the Secretary of Labor (SET) and the National Training Institute (INFOTEP) to focus more effectively on at-risk youth (estimated cost at appraisal US$ 2 million; actual cost US$ 1.6 million).

This component was to improve SET’s and INFOTEP’s capacity to reach disadvantaged out-of-school youth by strengthening SET/PJE program coordination, implementation, and evaluation functions, and to improve INFOTEP’s systems for course evaluation and supervision, as well as the gradual integration of PJE activities into INFOTEP.

Component 3: Strengthen and expand the "Second Chance" education programs Adult Basic Education (EBA) and Adult Secondary Education (PREPARA) (estimated cost at appraisal US$ 9.72 million; actual cost US$ 3.6 million).

This component was to strengthen and expand the Secretary of Education (SEE) primary and secondary equivalency programs (EBA and PREPARA) to address the needs of school dropouts who had an interest in continuing their studies. This included intensifying/accelerating programs and improving their quality, and improving the institutional capacity of the Department for Adult Education at the SEE to manage the expansion of Second Chance Programs.

Component 4: Support a pilot temporary employment program (estimated cost at introduction US$ 2 million; actual cost US$ 1.8 million).

This component was to pilot a temporary employment program targeted at selected municipalities in the province of Santiago to mitigate the effects on employment of the international economic crisis. The component was to provide temporary jobs for between 2,500 and 4,000 people over a period of 18 months.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project cost and financing:
Total project costs at appraisal were estimated at US$ 49.91 million, with the Bank to provide US$ 25 million, the Inter-American Development Bank (IADB) US$ 10 million, and bilateral agencies (unidentified at the time of appraisal) US$ 6 million. Actual project costs were US$ 33.2 million, with the Bank providing US$ 23 million and the IADB US$ 9.4 million. The ICR states that some bilateral financing had been expected but did not materialize; it does not identify the bilateral donors involved. Actual costs were lower than planned costs because of lower than expected demand for some course offerings.

Borrower contribution. The Borrower was to contribute US$ 8.91 million, but budget difficulties limited its contribution to US$ 0.8 million.

Dates. The project was approved on March 23, 2006.

The project underwent a first level restructuring in September 2010. This included a change in the PDO, reflecting the introduction of a fourth component, and a change in the environmental safeguards category (see Section 11). The restructuring introduced a number of changes to accommodate the addition of the temporary employment component, and extended the closing date by twelve months to June 30, 2012. Unrelated to the new component, the target value for teacher training under Component 3 was reduced at this time from 7,000 to 4,500.

A second level restructuring took place in 2012, extending the closing date for another twelve months (to June 30, 2013) to accommodate slow implementation of Component 3 and the consequent reallocation of resources from Component 3 to Component 1.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Relevance of objectives is rated high.

The project responded to a growing youth employment crisis in Dominican Republic, especially among poor and low-skilled youth. As a result of early school leaving and low enrollment rates in secondary education, a large number of youth were dropping out of the formal education system and finding limited opportunities to enter the active workforce. The Government was responding to the growing crisis with pro-poor initiatives, including investments in education and skills, and the articulation of a policy for youth development, underpinned by legislation, which continues to be a central theme of Government policy. The project objectives were also in line with the Bank's country assistance strategy. The project was an integral part of the FY2006-2009 CAS, supporting its emphasis on building social cohesion, including through enhancing the employability of people transitioning from school onto the labor market. This emphasis was continued in the FY2010-13 Country Partnership Strategy (a subsequent strategy has not yet been finalized), and is reflected in the introduction of the temporary employment component as an additional tool for improving employability.

b. Relevance of Design:

Relevance of design is rated substantial

Planned activities directly addressed the original PDO, including activities that were likely to improve the employability of at-risk youth. These planned activities reached beyond the scope of the original objectives, as the project came to support at-risk adult workers who also were beneficiaries of second-chance programs. Project design drew on the experience of a youth employment program financed by the IADB, introduced innovations relating to public-private collaboration for internships and life skills, and brought together key public institutions involved in workforce development.

The activities added to support the new objective, short term temporary jobs, corresponded less well as a cost-effective way to improve employability of poor, low-skilled, long-term unemployed people. Public works schemes generally have not been very successful in integrating the long-term unemployed into the labor market.


4. Achievement of Objectives (Efficacy) :

The original objective was to improve the employability of the Borrower’s poor at-risk youth population. It was to do so by: (i) building their work experience and life skills; and (ii) expanding second-chance education programs to complete their formal education.

The original objective was revised in 2010. The original objective and outputs were maintained, and the following objective was added: improving the employability of poor low-skilled long-term unemployed people between the ages of 18 and 65.

Efficacy is rated against both objectives.

Objective 1. Improving the employability of the Borrower’s poor at-risk youth population.

Efficacy is rated substantial.

Outputs

a. Building the work experience of the poor at-risk youth population

    • 38,000 youth at risk attended PJE-sponsored activities under the project between 2006 and 2013, exceeding the target of 32,000. 95 percent of participants came from the poorest areas, exceeding the target of 60 percent; and 58 percent were women, exceeding the target of 45 percent.
    • The National Institute for National Technical Training financed 400 training courses for youth at risk over the 2006-2013 period, meeting the target.
    • 3,000 private enterprises participated in the above activities over the 2006-2013 period by offering internships, exceeding the target of 2,000 enterprises.
    • New course offerings increased by 24 over the same period, falling short of the target of 40 new courses.
    • Links with the education system were strengthened, as 27 percent of PJE program participants registered in EBA or PREPARA within one year of program completion.
b. Expanding second-chance education programs to complete the formal education of poor at-risk youth
    • Participation in second-chance education programs in EBA increased by 5 percent (3,965 individuals) over the period 2006-2012, falling short of the target of a 12 percent increase (10,440 individuals). Participation in PREPARA increased by 61 percent (29,288 individuals) over the same period, exceeding the target of a 14 percent increase (6,720 individuals). 76 percent of EBA and PREPARA beneficiaries came from the poorest 40 percent of households, exceeding the target of 70 percent. 51 percent of beneficiaries were women, exceeding the 45 percent target.
    • Program drop-out rates decreased from 23 percent to 18 percent for EBA and PREPARA combined. There were substantial differences in drop-out rates between the two programs – 24 percent in EBA and 7 percent in PREPARA – with the latter showing better results in terms of enrollment and graduation.
    • 4,000 teachers were trained in adult education methodologies for the EBA over the 2006-2013 period, exceeding the target of 4,000. No additional teachers were trained for PREPARA, reflecting a government decision to focus on the basic education level.
    • Learning materials were developed for seven of the 12 modules for EBA, of which four already have been published.
c. Administration
    • An integrated management information system covering both EBA and PREPARA was put into place, tracking enrollments and attendance and pass rates, and conducting longer-term tracer studies.
    • The Secretary of Labor’s monitoring and evaluation system was redesigned to link payments with results.
Outcomes
    • 72 percent of PJE graduates have become employed or self-employed within six months after program completion, exceeding the target of 65 percent.
    • PJE had no significant impact on earnings, not reaching the target of a 10 percent increase compared to a control group.
    • 73 percent of the additional students drawn by the project into EBA and PREPARA obtained their basic or secondary degrees, respectively, falling short of the target of 80 percent. In the individual programs, graduation rates were 66 percent for EBA and 84 percent for PREPARA. No individual targets had been set for the project.
    • While there was no direct measurement of project effects on risky behaviors or changes in self-esteem, PJE graduates report improved perceptions about their situation and expectations for the future, and there are reported reductions in teenage pregnancy as a result of the life skills elements of the programs (ICR, page 13).
    • A number of additional positive effects were recorded in the impact evaluation of the PJE program:
      • In terms of employment quality, formal employment increased, particularly among women, who saw a 12 percent increase in the probability of having retirement benefits, health insurance, or a contract.
      • PJE graduates that are working appear to have greater job satisfaction than the control group, as indicated by a 3 percent reduction in the desire to seek a different job. This change may not, however, be statistically significant, and the training courses may have improved job satisfaction within existing jobs as well.
      • PJE had a positive impact on non-cognitive skills, where scores improved between 0.08 and 0.16 standard deviations with the program.
      • A positive impact was also found on alternative measures of life skills, particularly leadership skills, conflict resolution, self-organization, and persistency of effort.
      • PJE graduates were found to be more likely to participate in political or religious associations than their peers in the control group.
Overall, there was substantial progress towards achieving the objective of improving the employability of at-risk youth through the PJE, as indicated by 76 percent of participants finding employment within six months of completing training. Moreover, participation in programs also increased job satisfaction, life skills and well-being of participants. There is no indication that such was the case with the second chance programs.

Objective 2. Improving the employability of poor low-skilled long-term unemployed people between the ages of 18 and 65.

Efficacy is rated modest.

Outputs

    • 3,920 people participated in the temporary employment program over the June 2012 - June 2013 period, within the target range of 2,500 to 4,000 participants.
    • 98 percent of participants completed related life-skills training, exceeding the target of 80 percent.
    • 104 socio-economic projects were completed.
Outcomes

No outcome indicators on employability were specified, as the project was a temporary employment scheme. However, improved employability was specified as an objective without a target beyond temporary employment. Still, subsequent impact analysis drawing on M&E and administrative records from the Social Security Administration indicates that the formal employment rate of participants increased with respect to non-participants in the same geographic area. Post-program employment among the former rose moderately, by some 10 percent, and among the latter by some 3 percent. In part, these numbers, and especially those for non-participants, include the effects of the business cycle.


5. Efficiency:


Efficiency is rated modest.

The economic analysis in the PAD did not include quantitative analysis. It did discuss direct and indirect benefits in qualitative terms, and it drew attention to sector-specific cost effectiveness and cost benefit analyses in other projects and programs. It also discussed the fiscal implications of the project, although in general terms. The ICR (pages 15-16 and Annex 3) discusses economic efficiency separately for the PJE (68 percent of project costs), second-chance program (16 percent of project costs) and temporary employment (8 percent of project costs). For the PJE, the ICR estimates a rate of return of at least 5.6 percent, reflecting the fact that only a few of the benefits could be quantified. Additional, unquantified benefits are likely to be derived from lower crime, improved life skills, and access to health insurance. For second-chance education, while good data were not available, the ICR estimates a rate of return of 85 percent, with benefits derived from productivity gains due to the employability and productivity effects of improved curricula when spread over a large number of program participants beyond just the incremental 33,000 generated by the project. No formal economic analysis is undertaken for the temporary employment program. However, it is reasonable to assume that it had benefits at least equal to the wages paid temporary workers, as a result of enhanced life skills and socio-economic infrastructure produced under the temporary works. Overall, there is little evidence of public works being a cost-effective way of increasing the employability of beneficiaries. No fiscal/financial analysis was undertaken for the project.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
Yes
5.6%
68%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The relevance of the original objective is rated high: the objective addressed a central concern of the Government, and one that is in accord with the Bank's current CPS. The relevance of design under the original objectives is substantial: while planned activities went beyond the PDO, those activities would plausibly have been expected to contribute to achievement of the objectives. Efficacy is rated substantial recognizing that the PJE program had a positive impact on post-program employment and employability. Efficiency is rated modest, as there is little evidence of public works being a cost-effective means of increasing the employment of beneficiaries. The outcome under the original objective is rated moderately satisfactory.

In rating against the revised set of objectives, the ratings remain the same, due to the small size of the additional component (eight percent of project costs). Consequently, the outcome under the revised set of objectives is also rated moderately satisfactory

The weighted value of the outcome under the original objectives is the outcome rating is calculated as follows: (4 for moderately satisfactory) times (43 percent of total disbursements before the introduction of the additional objective), that is, 1.72. The weighted value under the formally revised objectives is 4 (MS) times 57 percent (disbursement), that is 2.28. The weighted average score is thus 1.72+2.28 = 4, which is moderately satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

    The changes that were introduced to achieve the original PDO continue to be supported by two new operations: a Bank-funded Flexible Employment System Project (P131109) under preparation; and an IADP project supporting the national employment system. Second-chance programs are expected to play an important role in supporting a new presidential initiative to combat illiteracy. With regard to the temporary employment program, it is still uncertain whether project-financed activities will be expanded. However, the Ministry of Labor now has the institutional capacity to do so if the economic situation warrants it.

    a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:

The project was strategically relevant, supporting the Government’s youth development strategy. The revised project was also relevant as a means of addressing a specific crisis. The Bank team drew on lessons learned from previous similar initiatives in the Dominican Republic and neighboring countries, bringing together public and private actors, and introducing internships and life skills training. The project had a coherent results framework and introduced a robust M&E system with a randomized experimental evaluation. However, there were shortcomings. The Bank team misjudged the challenges posed by weaknesses in second-chance programs. It did not sufficiently analyze the fiscal implications of the project, which resulted in implementation delays due to insufficient allocation of counterpart funds.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

The focus of the supervision effort was on supporting the development of workable processes. That said, the team provided exposure to international practice through study tours and technical discussions. It was actively engaged in implementing and using the M&E system to monitor progress and evaluate results. Some procurement issues arose (see Section 11b) that at least in part might have been picked up through the supervision effort. At restructuring, the temporary employment scheme was given an unnecessarily ambitious objective (improved employability), and its results framework failed to specify any outcomes.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

Government ownership and commitment to both objectives was strong. This commitment was reflected in Government policy and continued support from the main implementing agencies: the Ministry of Labor, INFOTEP, and the Ministry of Education. During implementation, the country was negatively affected by the international economic situation, and the Government collaborated well with the Bank to use the project to address employment shocks. A difficult fiscal situation throughout the project period led to insufficient budgeting for counterpart funds, slowing down project implementation.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

The project coordinating unit in the Ministry of Labor proved effective in managing a multi-agency project with private participation, and it was able to monitor implementation progress and results. INFOTEP performed effectively in implementing the youth development program, and the implementing agency for the temporary employment program was effective in building a public works program on short notice. Difficulties arose in the Department of Youth and Adult Education of the Ministry of Education, where frequent changes in staffing were a contributing factor to the modest performance of the EBA and PREPARA programs. Those difficulties resulted in procurement problems in the Ministry of Education (see Section 11b); likewise, procurement problems arose in the Ministry of Labor’s project coordinating unit due to the lack of a dedicated procurement specialist.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The project included a management information system for the PJE that incorporated all key participants, as well as the Ministry of Education’s school management system for second chance programs. M&E design also included a randomized experimental evaluation supported by household surveys of PJE graduates.

The results framework for the original PDO linked project components to outputs and final outcomes. Accompanying key indicators were measurable, with baselines and milestones set for monitoring project implementation. Results were based on impact evaluations. The results framework for the added objective did not include specific targets or indicators of change in the participating individuals and their families. Still, efforts were made to measure the impact on employment opportunities of participants.

The original PDO was supported by six key performance indicators set out in the PAD:

    1. At least 65 percent of graduates from the Government’s PJE (Youth and Employment) program are employed or self-employed six months after program completion.
    2. PJE graduates earn on average 10 percent more than the control group.
    3. The percentage of students in the EBA (Basic Education for Adults) program that drop out of the program is reduced from 23 to 18 percent.
    4. At least 60 percent of PJE beneficiaries and 70 percent of EBA and PREPARA (Secondary Education for Adults) beneficiaries come from priority areas I and II as identified in the Poverty Map for the Dominican Republic.
    5. At least 45 percent of program beneficiaries are women.
    6. At least 20 percent of participants of PJE are registered in EBA or PREPARA one year after program completion.

Indicators to measure selected externalities – such as reduction in risk-taking behavior, increased self-esteem, and building social capital – generated by the PJE and Second Chance education programs were to be developed and determined through an impact evaluation financed under the project.

An additional key outcome indicator was introduced when the additional objective was added. This indicator did not actually measure the improvement in employability specified by the objective;

7. Between 2,500 and 4,000 beneficiaries participate in the Temporary Employment Program (TEP).

b. M&E Implementation:

Implementation of M&E was done with an inter-disciplinary team from the Ministry of Labor, the IADB, and the Bank. The household survey was complemented by a series of studies that focused on specific qualitative aspects of the programs, such as perceptions of programs, providers, employers, and so forth. In addition, the Bank carried out a complementary evaluation on the relative merits of life skills training versus traditional technical training.

a. M&E Utilization:

The M&E system was used for monitoring progress towards achievement of outputs and outcomes and to evaluate implementation for eventual adjustments. The experimental evaluation formed the basis for assessments of overall impact of interventions and to draw lessons for future initiatives.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

The original project triggered no safeguard policies. However, with the introduction of the Temporary Employment component, the project’s safeguard category changed from C to B. The physical works to be undertaken under the component triggered OP 4.01 Environmental Assessment. The project’s compliance with the Environmental Management Framework was satisfactory, and minor weaknesses that were identified during implementation were subsequently corrected.

b. Fiduciary Compliance:

Financial Management. Financial management performance was moderately satisfactory, mainly as a result of weak performance under Component 3, including the closing of the Ministry of Education’s Special Account in 2008 due to inactivity. Financial audits for the last fiscal year were conducted on implementing entities and did not identify significant shortcomings. The last external audit for the Project has been completed. The opinions in the Management Letters were “unqualified” with regard to the project’s financial statements.

Procurement. Performance in terms of procurement was moderately satisfactory, mostly as a result of weaknesses in the application of the procurement guidelines both in the Ministry of Labor and the Ministry of Education, as well as a failed competitive process for the development and publishing of textbooks under Component 3. These weaknesses were due to lack of a dedicated, full time procurement specialist in the Ministry of Labor’s project coordinating unit, and low procurement capacity and high staff turnover at the Ministry of Education.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Negligible to Low
Negligible to Low
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

These lessons are drawn from the ICR (pages 21-22) and adapted by IEG:

Specific project design features play a role in determining program efficacy, but exogenous factors such as the level of economic activity and the structure of labor markets are also likely to play a role. Ultimately, this type of program can help better align the supply and demand of labor skills, albeit within the envelope determined by exogenous factors.

There are trade-offs between equity and efficiency objectives in skills development programs such as PJE. Targeting the most vulnerable, particularly poor at-risk youth who may have less access to jobs than their peers from other socio-economic groups, is a priority from an equity perspective. But the impact on labor outcomes among this group may be limited given the short nature of the training provided and the often dramatic deficiencies in skills and education among this population group.

Focusing on soft skills can improve effectiveness of training programs. Both employers and trainees appear to value these soft skills more than technical skills. Results from the project's impact evaluation suggest that training in soft skills might be more effective, in terms of employability, than technical training.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR offers good evidence combined with analysis in key sections assessing performance and outcomes. It is internally consistent, and mostly follows OPCS guidelines. On the weak side are its insufficient description of project costs and financing; and the assessment of the temporary employment program on the basis of temporary jobs created, rather than the employability of participants

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions