Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Third Basic Education Quality Improvement Project


  
1. Project Data:   
ICR Review Date Posted:
02/10/2014   
Country:
Uruguay
PROJ ID:
P070937
Appraisal
Actual
Project Name:
Third Basic Education Quality Improvement Project
Project Costs(US $M)
 56.0  137.56
L/C Number:
L7113
Loan/Credit (US $M)
 42.0  71.9
Sector Board:
Education
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  04/25/2002
 
 
Closing Date
12/31/2007 12/31/2012
Sector(s):
Primary education (54%), Pre-primary education (20%), Tertiary education (12%), Central government administration (11%), Other social services (3%)
Theme(s):
Urban services and housing for the poor (23% - P) Rural services and infrastructure (22% - P) Education for all (22% - P) Vulnerability assessment and monitoring (22% - P) Participation and civic engagement (11% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Eluned Schweitzer
Judyth L. Twigg Lourdes N. Pagaran IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Legal Agreement of June 17, 2002, the objective of the Third Basic Education Quality Improvement Project was "to increase the equity, quality and efficiency in the provision of preschool and primary (i.e., ages 4 through 11) education in Uruguay." This was to be achieved through: "(i) the expansion of the full-time school model, focusing on students from socio -economically disadvantaged backgrounds; (ii) the enhancement of the teacher training system and the introduction in the classroom of new teaching and learning instruments; and (iii) an increase in the efficiency of primary education institutions" (Schedule 2, page 12).

    The wording of the Project Development Objective on page 3 of the Project Appraisal Document (PAD) was shortened but is consistent in substance with the legal agreement: "to increase equity, quality and efficiency in the provision of preschool and primary education. These objectives will be achieved by: (i) expanding the full-time school model, which focuses on students from socio-economically disadvantaged backgrounds; (ii) improving the quality of preschool and primary education by enhancing the teacher training system and introducing new teaching and learning instruments in the classroom; and (iii) increasing the efficiency of education institutions."

    The project was re-structured four times (see Section 2d). The Project Development Objective (PDO) remained the same. However, in 2006 the project's focus on expanding Full Time Schools (FTS) to rural areas was broadened to include students in non-FTS time schools in urban areas. Changes included reducing the outcome targets from 86,000 student places in Full Time Schools to 47,000 places, and the introduction of an alternative model of school to address those disadvantaged children in non-FTS.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

This project was the third in the series "Basic Education Quality Improvement Projects," through which the World Bank supported the education strategy of the Government of Uruguay. Despite high primary completion rates in Uruguay, there were still issues in the quality of education for disadvantaged students. In order to address this issue, the Government had put in place a system of Full Time Schools (FTS), designed to channel resources to disadvantaged students. The Task Team Leader explained that the FTS model includes: extending the number of hours in the school day from about 4.0 to 7.5; introducing a new teacher development program; improving educational infrastructure; providing meals and nutritional snacks; and introducing children to a broader curriculum. Other features of the model relate to pedagogical approaches, including the promotion of an overall school environment with clear agreed norms within a supportive atmosphere. The model not only addresses learning achievement but reducing repetition and violence in urban contexts. At the preschool level, coverage among 5 year olds is 95.5%, while coverage among 4 year old children is 94.5% across all schools in Uruguay. In FTS preschool children have an extended day, improved curriculum, and other interventions.

A 1999 Student Assessment for 6th graders indicated that disadvantaged students in the FTS program at that time did better than those in regular schools, indicating that the FTS system had a positive impact on learning outcomes for those students. The Government thus decided to expand and improve the FTS system in order to further target disadvantaged students, and the approach is now part of the overall government education strategy rather than a project-specific initiative.

Within this context, the project had two components:

Component 1. Expansion of the Full-Time School Model (US$44.52 million at appraisal, US$120.47 million at closure). The aim was to expand an existing system of Full-Time Schools (FTS) and introduce innovations to make the system more effective. This component had four sub-components:

    • construction, transformation, and rehabilitation of full-time schools and the institution of school maintenance funds to prevent the deterioration of public assets;
    • acquisition of school equipment, learning materials, and school libraries;
    • teacher training on the full-time school model, including support for disadvantaged students, new educational technologies, and bilingual education. and
    • strengthening partnerships between schools and parents.

Component 2: Institutional Strengthening (US$11.06 million at appraisal, US$16.05 million at closure). This component aimed to strengthen both individual schools and the education system as a whole. It had four sub-components:
    • reforming the system of teacher training;
    • the development and implementation of Education Improvement Projects (PMEs);
    • monitoring and data collection in the education program; and
    • project administration.

Post-restructuring, the components remained the same, but the focus of activities changed, primarily broadening the type and number of schools involved in the project and changing from reaching the disadvantaged in rural schools to impacting most schools with disadvantaged children where the most extreme poverty and violence exist, rural or non-rural (known as Priority Attention Schools). These are referred to in this review collectively as non FTS, the term used in the ICR. A number of activities were scaled up and/or mainstreamed in 2006 and therefore were no longer financed under the project. In its final years, the project concentrated on in-service teacher training and infrastructure activities.

2006 Changes to Project Activities

Component 1.
    • the provision of technological equipment, textbooks, and learning materials was expanded to include regular schools that serve disadvantaged children;
    • teacher training activities were expanded to teachers and staff in all schools with disadvantaged students;
    • bilingual teacher training was discontinued under the project, but scaled up and institutionalized under the Pre-School and Primacy Education Council (CEIP) umbrella;
    • training of teachers on new technologies was discontinued under the project and mainstreamed into the regular education program; and
    • management information system strengthening activities were completed and therefore did not continue after 2006, with the exception of the international learning assessments.

Component 2.
    • construction of a teacher training college; and
    • scale-up of the number of education improvement subprojects, with emphasis broadened from nutrition to pedagogical and other inputs.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Costs: Construction costs rose considerably over the course of the project. This increase was due to a change in the Government strategy from classroom rehabilitation to new construction, and significant increases in construction costs in the country. Construction costs rose on average by about 144% during the period 2002-2009. The Uruguayan National Statistical Institute Construction Cost Index reported an increase in construction costs from August 2006 to June 2009 of about 27% (Project Paper on Additional Financing, para.13, page 3 ).

Financing:

    • January 21, 2004: Schedule 1 was amended to increase the disbursement percentage for imported goods and reallocate amounts under existing categories of expenditures.
    • February 24, 2010: Additional Financing in the amount of US$29.9 million was approved to address a financing gap that had arisen due to increased project costs.
    • June 6, 2012: Funds were reallocated to finance an expansion of the in-service teacher training program and for the hire of additional advisors for the supervision of works.

Borrower Contribution: The Borrower contribution was estimated to be US$ 14.0 million at appraisal. As per the ICR, the contribution was US$65.12 million, or 465% of the appraisal estimate, as the Government mainstreamed and assumed responsibility for several key project interventions.

Dates:
    • July 13, 2006: The project was re-structured to include an extension of the closing date from December 31, 2007 to December 31, 2009, to allow for expansion of activities.
    • February 24, 2010: Additional Financing was approved with a closing date of December 31, 2012.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Relevance of Objectives is rated Substantial. At appraisal in 2002, the project objective was relevant for the Country Partnership Strategy of 2000, and it remains highly relevant to the current World Bank Country Partnership Strategy (2010), which supports the "Government's efforts to increase equity, quality, and efficiency in the provision of preschool and primary education." At appraisal the objectives were also highly relevant to the country's economic context, where education was almost universally available but where there were specific issues in addressing the quality of education for disadvantaged groups. However the measurability of the first and second objectives, as articulated, was challenging given the lack of clarity in measuring results ( see section on M&E).

The project is also highly relevant to the current priorities of the Government of Uruguay and its education sector reform program, which prioritizes preschool and primary education. The program has received the support of the President of Uruguay. The objective of expanding the FTS approach is currently being accelerated, with 50% of funding coming from the follow-on World Bank-funded operation "Support to Uruguayan Public Schools Project," approved in September 2012, and approximately 80 new schools anticipated over the next four years.

b. Relevance of Design:

Relevance of Design is rated Substantial. Project design was substantially relevant to its objectives, as the components were logically and plausibly linked to the three project development objectives. Inputs to Component 1 were an expansion of activities supporting the Full Time School model. Component 2 contained activities that supported institutional strengthening, chosen to help improve efficiency within the education system (improved teacher training, data collection, and increased local level and community engagement through school projects). Thus the logical flow of the results framework was commensurate with the objectives to be achieved.


4. Achievement of Objectives (Efficacy) :


Increase equity in the provision of preschool and primary education by expanding the full time school model: Substantial

The original focus on the Full-Time School model was broadened over the life of the project so that inputs also impacted Quintile I and II students in non-FTS. Additional information was provided by the Region on students in non-FTS. The Region also noted that inputs to FTS and non-FTS uniformly affect pre-school programs.

Outputs:
    • 868 new FTS classrooms were built (329 in new schools, 286 new classrooms in existing schools, and the remainder rehabilitated) in urban areas. This is less than the original target of 1,356, but slightly over the 850 target of December 2012.
    • 14 FTS rural schools were rehabilitated (discontinued as a target at re-structuring in 2006, after policy changes). According to the Borrowers' comments, 45 FTS were added between 2010 and 2012, and the number of FTS rose from 92 in 2002 to 175 in 2012.
    • The Region provided additional information that 207 schools received improvements to pre-school facilities as well other quality-related inputs, providing 16,400 disadvantaged and most disadvantaged students with access to an improved preschool education.

Outcomes:
    • The number of students in FTS rose from 21,419 in 2002 to 40,156 in 2012. This indicator was partially met, with the original target of 86,000 students revised to 47,000 students in 2012. The achievement below target was the result of the Borrower's decision to limit class sizes to 25 students. In addition, the shift in focus from FTS only to non-FTS schools with a high majority of low income students meant that the indicator does not accurately reflect the overall impact of project activities.
    • The number of students from the first and second income quintiles as a percent of all FTS students rose from 47% in 2002 to 52.7% in 2012, only a small increase. The FTS share of all students in the first and second income quintiles more than doubled between 2002 and 2011, from 7.4% to 15.4%.
    • The proportion of FTS students as a percent of total urban enrollment in preschool and primary almost doubled, from 6.2% to 11.7%, between 2002 and 2011.
    • Access to preschool for children in Quntiles I and II increased from 70% aged 4 and 90% aged 5 in 2002 to 87% and 92.7% in 2012. 62% of these children are enrolled in the FTS and PAS schools. Results from the OECD 2009 PISA tests demonstrate that in Uruguay students that attend pre-school score on average 50 points higher than those who have not.
    • The percentage of sixth grade students from disadvantaged and very disadvantaged backgrounds who performed at a satisfactory level changed between 1999 and 2009 as follows: in FTS only, increased from 53.8% to 64.8% for Language, but decreased from 35.6% to 33.4% for Math. For all disadvantaged/very disadvantaged students (FTS plus non-FTS), the performance was better: an increase from 48.8% to 63.1% for Language, and an increase from 27.9% to 31.8% in Math. There was a change in testing methodology in 2005, but presumably that change would have similarly impacted FTS and non-FTS students. It would appear from these data that the FTS system had little discernible advantage over the non-FTS system for disadvantaged students. However, without knowing more about student characteristics in these two systems, it is not possible to say more at this time.
    • The equity gap -- defined as the difference in the percentage of students from advantaged socioeconomic backgrounds performing in a satisfactory manner, and the percentage of students from disadvantaged backgrounds performing in a satisfactory manner -- changed between 1999 and 2009 as follows: for disadvantaged students, decreased from 37.9 to 23.1 for Language and 40.2 to 38.3 for Math; and for very disadvantaged students, decreased from 39.2 to 31.9 for Language and increased from 42.0 to 46.4 for Math.
    • Further analysis is needed to clarify whether the complete package of inputs to the FTS model produces learning outcome results that are better than those in the non-FTS of various types, as there are many variables involved.

Improve the quality of preschool and primary education (ages 4 to 11) with a focus on the socio-economically disadvantaged and very disadvantaged contexts: Modest

Outputs:
    • Equipment for 868 FTS classrooms was purchased. This was slightly above the 2012 target of 850, and above the original target of 694 classrooms. 175 FTS received computers and equipment,
    • Video equipment was provided to 34 FTS,
    • 280 sets of learning materials were distributed to FTS and Priority Attention Schools (PAS),
    • 40 sets of bilingual materials and one set of library books were procured for each FTS,
    • One textbook per child in 1st and 2nd grades was produced and distributed.
    • 72 FTS and PAS schools received inputs to strengthen links between schools and communities, including social supports for children experiencing problems.
    • Compensation was increased for 60 teachers who were reclassified (the ICR does not state how they were reclassified) in 2006.
    • A new teacher training institute was constructed, and books and new teacher training materials were provided to all teacher training institutes.
    • A new pre-service teacher training curriculum was designed.
    • 2,785 education personnel were trained in the FTS model, and 2,100 were trained in PAS. 415 teachers were trained in bilingual education.
    • An in-service training system was developed.
    • There was national participation in international assessments (Trends in International Mathematics and Science Study and Progress in International Reading Literacy Study).
    • School construction was developed in an innovative fashion, with input from the communities resulting in innovative school designs adapted to the local environment. This was designed to support the overall goal of community engagement in education.

Outcomes:
    • The percentage of sixth grade students from disadvantaged and very disadvantaged backgrounds who performed at a satisfactory level changed between 1999 and 2009 as follows: in FTS only, increased from 53.8% to 64.8% for Language, but decreased from 35.6% to 33.4% for Math. For all disadvantaged/very disadvantaged students (FTS plus non-FTS), the performance was better: an increase from 48.8% to 63.1% for Language, and an increase from 27.9% to 31.8% in Math. Some of these results may not be statistically significant.
    • The data presented on achievement for students from disadvantaged backgrounds and on the equity gap run from 1999 through 2009, while the project's time period was 2002 through 2012. It is therefore uncertain that the outcomes presented here can be attributed with confidence to the project's interventions. .The Region provided additional information on repetition rates pointing to the majority of improvements occurring under the project period. No learning assessment has been held since 2009.
    • It may require more time for the project's inputs to fully impact disadvantaged student populations in the FTS system. There are no data in the project for other issues the FTS system is designed to address, such as violence in urban areas.

Increase the efficiency of the preschool and primary education system with a focus on socio-economically and disadvantaged and very disadvantaged contexts: Substantial

Outputs:
    • The achievement of this PDO is linked to outputs listed under the previous two objectives. Improvements in access and better staffed and equipped schools would have affected retention and completion rates for the disadvantaged. The Education Improvement Projects increased community engagement in schools, a known factor in creating a supportive and more efficient learning environment.
    • 320 computers and 120 printers were provided, and 2,600 personnel were trained in data collection and the use of equipment.
    • 912 Education Improvement Projects (PMEs) were developed (not meeting the target of 1,100 PME). This activity was integrated into the regular education system program.

Outcomes:
    • The Ministry of Education now has a functioning data collection system able to provide national level data for the education system and the project, although more needs to be done to effectively measure learning achievements.
    • Repetition rates improved, decreasing in the FTS for the lowest two income quintiles for first grade from 23.2% in 1999 to 12.8% in 2012, surpassing the target of 15%; and for second grade from 17.1% in 1999 to 6.1% in 2012, surpassing the target of 10%. For all schools (FTS plus non-FTS), there was also improvement, but not as strong as for the FTS alone: a decrease in repetition of first grade from 23.4% to 18.5% from 1999 to 2012, and a decrease in repetition of second grade from 17.6% to 9.7% over the same time period. Data presented by the Region demonstrate that the majority of these improvements took place between 2002 and 2009. These improvements can partly be attributed to the effect of improved pre-school access and quality.
    • Drop-out rates for children in grades one through six in the first and second income quintiles decreased from 2.1% in 1991 to 1.0% in 2012 for the FTS alone, and from 2.2% to 1.5% in all (FTS plus non-FTS) schools.
    • Poor attendance rates for children in grades one through six in the first and second income quintiles decreased from 9.8% in 1999 to 8.3% in 2012 for the FTS alone, and from 10.0% to 9.8% in all (FTS plus non-FTS) schools.

5. Efficiency:

Efficiency is rated Modest. The Economic Analysis in the Project Appraisal Document calculated a possible Economic Rate of Return of 37% (with the minimal return being above 13%) on activities in Component 1 of the project, which focused on expanding the FTS model (largely construction). This seems high. The Cost Benefit Analysis carried out for the PAD noted the possibility of increased indirect costs over time due to the expanded salaries and infrastructure created for the FTS. The Economic Analysis noted two major benefits from the expansion of the FTS model, covering the period 2002-2035:

1. Students would benefit from improvements in the quality of their education which would translate into better qualifications and improved earnings in the labor market. Over the long term this should result in an increase of human capital per year of schooling and a more efficient use of education funds.
2. In addition the FTS model had an impact on the labor force participation of primary caregivers who could work longer hours and contribute to their income and to the economy overall.

When recalculated at the time of the 2009 Additional Financing request, the ERR was estimated to be 15%,with additional financing, taking into account a lower number of beneficiaries had originally been anticipated. The ICR bases it's overall economic analysis on that carried out for appraisal of the follow-on operation. This gives an ERR of between 7.4% and 9.6%. It is not entirely clear how this analysis is accurate for the project under review; although they both support the FTS model, project components are not the same. The ICR also undertakes a financial analysis which notes that higher education costs as a result of the increases in salaries and training have been financed by increases in the education budget. It makes an assumption that this will not be a burden on the State budget, as average per-student spending in primary and secondary schools is 7.2% as opposed to a regional average of 13.1%. This is not a valid assumption, as there are many variables between countries and systems. The ICR calculates an IERR between 7.3% and 9.6% based on assumptions of improvements in lifetime incomes for students.

Construction costs under the project rose on average by about 144% during the period 2002-2009. The Uruguayan National Statistical Institute Construction Cost Index reported an increase in construction costs from August 2006 to June 2009 of about 27% (Project Paper on Additional Financing, para.13, page 3 ). This resulted in considerably higher than anticipated overall project costs. This was partly because the ratio between rehabilitation and construction costs was reversed (70/30 to 35/65%) by the end of the project. Additional information provided by the Region reinforces these points, also noting that the decision to shift from classroom rehabilitation to new construction meant that there was a long-term benefit to the system in terms of the efficiency of new buildings and lower maintenance requirements in the new schools.

NOTE : The ICR estimate below is based on the estimated coverage at appraisal for Component 1 plus the additional financing.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
37%
79.5%
ICR estimate:
Yes
7.4%
273%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Relevance of objectives and design is rated Substantial. The objectives were chosen to target a key issue in improving education in Uruguay, the status of disadvantaged students. That issue remains substantially relevant to the country context, Bank strategy, and Government strategy. The project's planned activities were directly and plausibly aligned with achievement of those objectives. Achievement of two of the project's objectives is rated Substantial, while one is rated Modest. Project activities substantially impacted the equity of education in terms of access and results. However, the inputs to improve quality as judged by learning outcomes show only a modest improvement, particularly for Mathematics. The objective to improve the efficiency of education was substantially achieved, as there was improvement in repetition rates for disadvantaged students. The project's efficiency is rated modest, as costs rose dramatically over the project's lifetime, and the ICR's calculations of economic rate of return are based on assumptions that may not be valid. Taken together, these ratings are indicative of moderate shortcomings and therefore an outcome rating of Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

    Government support for this project was and remains high. However, the project addressed the educational needs of the disadvantaged and most disadvantaged, and although progress is being made, it is particularly difficult to address the learning needs of this group in any country, particularly in urban areas. Sustained inputs to the quality of the FTS system will be necessary to ensure that this group succeeds, as the education system is only one of the variables affecting the learning achievements of these students.

    a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:
The project received a satisfactory Quality at Entry Review rating in June 2002. A Quality Assurance Group (QAG) assessment noted that the project was in line with the strategic priorities of the Government, Uruguay's Education Reform Program of 2000, and the Country Assistance Strategy of the World Bank (also 2000), which focused on addressing pockets of poverty. The project was part of a long-term support strategy focusing on basic education; this strategy spanned almost two decades, and this was the third of four planned projects. The project incorporated lessons learned from previous projects, stakeholder workshops, and an impact study of the FTS model conducted in 2002. The Government's involvement in project preparation was high, as it supported the FTS model, which was nationally devised and owned.

However, the ICR notes a number of shortcomings at entry: (a) project preparation was accelerated, and the project was appraised while the previous project (MECAEP II) was still under implementation with undisbursed funds, and when preparation work for teacher training reform and other components was incomplete; and (b) project preparation did not mention the macroeconomic risk of the 2001-2002 regional crisis in Argentina and Brazil, although this crisis was ongoing at the time of preparation. These shortcomings led to a slow start-up and difficulties with counterpart funding. In addition, there were issues with M&E (see Section 10).

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

There were three Task Managers over the ten-year course of the project, reflecting internal job rotation within the Bank during that time period. Project implementation performance was rated unsatisfactory between 2005 and 2007. The reasons for this rating and the actions the supervision team took to address the issues are not fully clear from the ICR, but the lack of counterpart funding during the economic crisis may have contributed. A mid-term review was held in 2005. Given downgraded ratings at that time, the 2006 restructuring was timely and appropriate to ensure improved progress. The ICR notes a QAG review of supervision, but it gives no further information on the outcomes of this review. The supervision team was in regular contact with the Project Coordination Unit and relevant education departments and provided inputs on both implementation and policy to the Ministry when needed. However, issues with the M&E framework were not addressed early enough in the project to make corrective actions.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The Government has been a motivated partner throughout the project, and the project's objectives have remained unchanged through three administrations. The economic crises in Brazil and Argentina in 2001-2002 affected the economy of Uruguay, resulting in slowed economic growth and an increase in the number of people in poverty and the number of disadvantaged students. Counterpart funding was limited between 2002 and 2006 due to public spending quotas, slowing down disbursements and implementation. The project was also competing for counterpart funds with the preceding operation until February 2004. These externalities impacted project implementation but were mostly beyond Government control. There have been no recent issues with counterpart funding.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:
Government agencies involved in implementation were highly supportive of and engaged in the project. The Project Coordination Unit (PCU) was well and continuously staffed and was respected within the education system. Coordination within the PCU and between the PCU and line agencies was good. The mainstreaming of project activities deemed successful into the relevant departments indicates the extent to which project activities were owned by the education system in Uruguay. There were, however, issues with data collection, particularly with learning assessments, that could have been avoided . More effective data collection and analysis would have enabled better monitoring and evaluation.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

M&E design included strong PDO level indicators. However, there were discrepancies in the initial documentation, making it unclear what exactly was to be measured (children at Full Time Schools, or all disadvantaged and very disadvantaged students). The results framework contained 24 intermediate outcome indicators, many of which were simply outputs, such as the number of computers and software sets distributed. There were no real indicators for the success of the institutional strengthening activities or for preschool improvement, which is stated as part of the Project Development Objective. Similarly, the indicator for the Education Management Information System (EMIS) simply itemizes the equipment for the EMIS system rather than measuring whether or not the system is functioning. A number of indicators relate to the FTS system rather than to the broader system for addressing disadvantaged students.

The M&E plan contained provisions for nine evaluation studies. These included studies of the partnership between schools and parents, the Education Management System, the pre-service teacher training, the in-service teacher training, the bilingual education program, the teaching of math and English, and a number of learning assessments. Two reports on the status of education in Uruguay were also planned.

b. M&E Implementation:

Monitoring and Evaluation was carried out by the Project Coordination Unit and line agencies. The Preschool and Primary Education Council (CIEP) monitored the routine performance of the individual FTS through its regular administrative structure. All the studies were completed.

There were changes to some of the evaluation tools in the results framework during the life of the project. For example, the National Evaluations of Learning Achievement carried out in 2002 used "classic test theory," but the 2005 and 2009 tests were based on "item response theory," limiting their comparability and affecting assessment of achievement of one of the PDO indicators. In 2010 there was also a change in the methodology used to characterize socioeconomic vulnerability for efficiency statistics, including the PDO indicator addressing repetition rates. Quintiles were only used to measure socioeconomic vulnerability starting in 2010. This necessitated re-calculating data from 2002 to 2010 for the purpose of the ICR. The planned assessments and M&E studies were all carried out. However, sound data on student academic performance are still lacking, although the country did participate in two international assessments,. The absence of concrete data on learning achievement over the life of the project makes it difficult to evaluate project impact.

A number of the targets were changed over the life of the project. At re-structuring in 2006 and in other restructurings, targets for some of these indicators were revised downwards or were changed due to policy changes affecting project inputs. However, no breakdown of urban/rural students was included when the project was restructured and the focus on urban areas added, nor are there any indicators or proxy for possible reductions in urban violence and other issues mentioned in the ICR as behind the FTS initiative.

a. M&E Utilization:

Results from the studies carried out under the project were disseminated and used to improve project activities. The Preschool and Primary Education Council (CEIP) used data to improve education services; for example, it added a new math module to the teacher training system as a result of lower scores on the 6th grade learning assessments.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
The project was upgraded from Category B to Category C during preparation. The QAG assessment commended the quality of the work done on the Environmental Assessment, and the implementation of the Project Environmental Plan was satisfactory, indicating compliance with safeguard policies. New school buildings and classrooms were built on public land, and only one case of an individual who was resettled and needed compensation occurred.

b. Fiduciary Compliance:

    The ICR reports that project financial management was sound, as the PCU had extensive previous experience. Risks related to the adequacy of financial management staff, systems, and software were identified in the PAD and addressed. The ICR (p. 8) states that there were minor financial management shortcomings during implementation, but it does not provide details of these. Procurement plans and arrangements were satisfactory, as the PCU already had experience in Bank procedures.

c. Unintended Impacts (positive or negative):

The FTS model has generated interest in other countries in the region, including the Dominican Republic and Bolivia, to whom the Government of Uruguay has provided technical assistance.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Satisfactory
Moderately Satisfactory
Project preparation did not take into account the risks stemming from economic downturn in the region, and there were shortcomings in the monitoring and evaluation framework that led to later problems. 
Borrower Performance:
Satisfactory
Moderately Satisfactory
Although implementation was thorough overall, data collection issues, particularly on learning achievement, have hampered project evaluation. 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The ICR (pp. 17-18) contains lessons that have both broad global application and project/country-specific application:

1. Incremental implementation over time contributes to sustainability. The lesson learned here is that the Government of Uruguay's long-term investment in the FTS model led to its consolidation as part of state policies. IEG notes, however that the need for this model had to be adjusted during the project due to cost and other considerations also emphasizes the need to be thorough in costing and project design across both urban and rural environments. IEG agrees that innovations will inevitably grow and change over time in response to differing situations.

2. It is important to clearly align the operational definitions of the Key Project Indicators with a Project's PDO and target population. Project appraisal documents should include the methodology for calculating target baselines, which should be aligned with the stated PDO and target populations.

3. It is critical to ensure some comparability of learning assessment results over time. As assessments are always improving, it important to ensure that some results are comparable over time in spite of changes in methodology.

4. Ensuring the FTS are supported and maximized is important for their success. The ICR notes a number of lessons learned relating specifically to the FTS model, as sustaining quality programs requires ongoing inputs. These include improved targeting of very disadvantaged students, and further impact evaluations and evaluations of the costs and benefits of various parts of the FTS program.

14. Assessment Recommended?

Yes
Why?
This project is tackling a significant and difficult issue: quality education for the most disadvantaged in Uruguay. The FTS model, which is unique to Uruguay and contains many practices that are globally identified as being key to a quality education, is being exported elsewhere and has shown some improvements in retaining disadvantaged children in school. Many of its features are now standard within Uruguayan education and have been integrated into the education system. However, unless the program is monitored carefully and analyzed as to which aspects are working and which are not, it will be difficult to assess its overall impact. This is an important issue and also a potential opportunity for future expansion. An impact assessment process might assist in identifying ways to better capture Bank investment results and impact in terms of monitoring and evaluation relevant to the upcoming project and the overall Government program..

15. Comments on Quality of ICR:

The ICR is clear and concise, giving a good overview of the country and education context. It captures the relevance of the project to country needs and provides a detailed analysis of project components, the rationale behind the several restructurings, and project outcomes. It does not entirely capture the complexity of the education system in Uruguay as it affects Quintile I and II students, or provide information on pre-school activities and PAS students to strengthen arguments for improvements in equity.

The ICR could have also have provided more information about the impact of those activities that ended or were mainstreamed early in the project cycle, even though this is hard to capture over a ten-year implementation period. It is also not usual to use the data from an economic analysis for a follow-on project that might have different components and economic parameters, although in this case project activities were closely related.

The ICR team did do the necessary calculations to incorporate the use of quintiles that had not been in use at project preparation and does base its evaluations on data, although this is incomplete. The ICR rates project results both pre- and post the 2006 restructuring, which was not necessary given that the PDOs did not change and were not reviewed by the Board.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions