Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Basic Education Project


  
1. Project Data:   
ES Date Posted:
09/24/2004   
PROJ ID:
P009089
Appraisal
Actual
Project Name:
Basic Education Project
Project Costs(US $M)
 2,515.2  2,567.6
Country:
Turkey
Loan/Credit (US $M)
 300  286.2
Sector, Major Sect.:
Central government administration, Primary education, Tertiary education,
Law and justice and public administration; Education; Education
Cofinancing (US $M)
   
L/C Number:
L4355      
   
Board Approval (FY)
  98
Partners involved
 
Closing Date
06/30/2001 12/31/2003
         
Prepared by: Reviewed by: Group Manager: Group:  
H. Dean Nielsen
John R. Heath Alain A. Barbu OEDSG

2. Project Objectives and Components:

a. Objectives
The Project is the first phase of an Adaptable Program Loan (APL) in support of the Government of Turkey's Basic Education Program and its objectives. The Project objectives of APL1 as stated in the Loan Agreement (amended December 1999 and November 2000 as a result of factors beyond Bank and Borrower control (major earthquake) ) were: 1. The expansion of enrollments in Basic Education Schools;

2. Training of teachers, school principals, and inspectors;
3. Development of central and provincial implementation capacity to carry out the Program;
4. Creation of a mechanism to monitor and evaluate the outcomes of the Program;
5. Reconstruction and rehabilitation of educational facilities for preschool, primary and secondary education which were destroyed or damaged in the Marmara earthquake of August 1999, and
6. Resumption of preschool education, basic education, and secondary education in the area affected by the Marmara earthquake of August 1999.

b. Components
Project components in the Loan Agreement are somewhat different from those stated in the Project Appraisal Document and were, like Project objectives, amended in December 1999 and November 2000; thus this review will consider the components as stated in the amended Loan Agreement, as follows:
1. Expanded Basic Education Coverage (expected cost - US$48.4 million; actual cost - US$94.4 million);
2. Improved Basic Education Quality (expected cost - US$233.5 million; actual cost - US$190.4 million);
3. Implementation Support (expected cost - US$4.8 million; actual cost - US$0.6 million);
4. Monitoring and Evaluation (expected cost - US$13.3 million; actual cost - US$1.0 million);
5. Monitoring Response Facility (MRF) (expected cost - no information; actual cost - no information);
6. Marmara Earthquake Relief Component (expected cost - no information; actual cost - no information).

c. Comments on Project Cost, Financing and Dates
The Project was mostly funded by the Turkish Government, which committed $2,215 million (88%) of the Project's total budget of $2,515. The Bank loan (APL1) commitment was $300 million. By Project closing the Government's contribution totalled $2,268 (2% above commitment); disbursement of APL1 was $286 million (about 5 percent below commitment), $6 million of which was cancelled in the last project FY and $8 million at closing. Disbursement by component was significantly different from PAD estimations; for example, expenditures under Component 1 (Construction and Rehabilitation of school buildings) were 195% of those estimated at appraisal; under Component 2 (Education Materials), they were 161% of estimated costs; under Component 2 (In-service Training Program), 12% of estimated, and Component 4 (Monitoring and Evaluation), 10%. In the case of the last two, In-Service Training, and Monitoring and Evaluation, the deviation reflects Project shortfalls in program implementation and outputs; for the two major budget overruns (construction and materials), no explanation given in the ICR. The Project closing date was extended three times (totalling 2.5 years), mostly because of persistent procurement problems and the need for more time to deliver significant Project inputs, such as learning materials and Monitoring Response Facility grants to schools, but also due to the Marmara earthquake, political turbulence, such as a change in government, and in frequent changes in government regulations.


3. Achievement of Relevant Objectives:

The Project did not specify any specific performance (outcomes or impact) indicators for its objectives and none were used in tracking Project effectiveness, making evaluation of achievement difficult. Moreover, the triggers for moving to APL II were mostly input indicators. Concerning Objective 1 (achieved), Expansion of Enrollments, Government figures reported in the ICR do show an increase in the net enrollment ratio for the eight-year primary school (since 1998 the locus of compulsory basic education) from 86% in 1997 to 96% in 2003, mostly attributed to the Basic Education Program's success in expanding school places and recruiting new teachers. Output data for the relevant APL1 component show the number of Project schools rehabilitated, but do not provide
information on enrollment increases in target areas.

On Objective 2 (not achieved), Training of Teachers, Principals and Inspectors, the Project performed far below expectations: major teacher training on ICT was undertaken under the Government budget, but under the APL only a fraction of planned inservice training was provided (training fewer than 7500 teachers and administrators compared to more than 300,000 planned).

Regarding Objective 3 (not achieved), Development of Central and Provincial Implementation Capacity, the Project scope was limited to: a) providing consultant and material support to the Project Coordination Team (PCT), which was located outside the Ministry of Education (MONE) management structure, and b) improving the Ministry's management information system (MIS). The ICR found no evidence of implementation capacity improvement at the PCT as a result of consultant and material support, and the MIS parts of the Project were dropped.

For Objective 4 (partially achieved), Creation of a Mechanism to Monitor and Evaluate Outcomes of the Program, the Project had no specific plan for such a mechanism and no performance indicators; it did, however, support a successful national assessment of learning achievement (surveying around 70,000 from 500 schools); it also planned to sponsor 11 special studies (though the Ministry or university researchers), but only 3 were undertaken and completed.

For Objectives 5 and 6, no information was given.

4. Significant Outcomes/Impacts:

Despite natural disasters, political instability, and a difficult administrative environment (especially for procurement), the project was able to deliver a highly regarded package of learning materials to a large number of rural schools, reaching almost 3 million students, and, based on feedback under the ICR beneficiary assessment, this already seems to be having a positive affect on enrollment levels.
  • The Project was able to use the results of a qualitative study on school facilities to determine that closing small village schools and bussing students to larger ones in town was having an adverse affect on girls' attendance; it was flexible enough to reverse its position on bussing and amend the project to rehabilitate 1300 formerly closed small village schools.
  • The Government commitment to provide universal access to 8-year basic education schools is strong and backed by financial support. The Government's contribution to the APL exceeded the original estimate, and its tax incentives are encouraging an unprecedented level of private giving to education. Program sustainability is thus quite promising.

5. Significant Shortcomings (include non-compliance with safeguard features):

The APL was designed in support of the Government's Basic Education Program, but did not adopt broad Program objectives; instead it articulated a set of objectives which were really components; for these objectives no outcomes measures were specified, undermining the possibility for results-based project supervision and evaluation.
  • There was poor articulation between Project objectives and their related components: for example, the Objective on Training of Teachers and Supervisors is linked to Component 2, but that component, covering "Improved Basic Education Quality," is much broader in scope. Also, there was a major component, Monitoring Response Facility, a school-based innovation and action research program, which was associated with the wrong Objective (Mechanisms for Monitoring and Evaluation).
  • For some components (those dealing with Implementation Support and Monitoring and Evaluation) there were no output indicators, leaving no clear basis for supervision and evaluation.
  • The APL1 triggers for moving to the second phase, were mostly based on inputs (contracts signed and materials being delivered), and since many of these were fulfilled, the second Phase has gone ahead, even though the APL1 was found to be ineffective on most of its measurable outputs and outcomes.
  • Much of the initial Government concern leading to the Project was the low participation in Basic Education of children from poor and disadvantaged areas of the country like the Eastern and Southeastern Provinces and from urban slums, yet the Government and the ICR do not report any of the outcome data specifically for the poverty targeted groups. No conclusions can be drawn as to whether the Program has been successful in increasing participation and learning outcomes among the poor.
  • The Ministry of National Education (MONE) of Turkey was described as having a fragmented institutional structure, characterized by poor internal communications, yet the Project built its base -- and its institutional development efforts -- at a project implementation unit that stood outside of the Ministry and had poor communications with it. This was one of the contributors to poor institutional development under the Project.
  • According to ICR listings, there were several large gaps between Bank supervision missions (17 months, 19 months, and 13 months), which would not be expected for a high intensity ALP operating in a difficult administrative setting.
  • 6. Ratings:ICROED ReviewReason for Disagreement/Comments
    Outcome:
    UnsatisfactoryUnsatisfactory
    Institutional Dev.:
    NegligibleNegligible
    Sustainability:
    LikelyLikelySustainability of those parts of the project that were implemented appears likely overall, in the context of good Government ownership of the LT program.
    Bank Performance:
    UnsatisfactoryUnsatisfactory
    Borrower Perf.:
    UnsatisfactoryUnsatisfactory
    Quality of ICR:
    Satisfactory

    7. Lessons of Broad Applicablity:

    Triggers for APL continuation should be policy-oriented and results focused, and not just focused on procurement actions; lessons learned in one APL phase should be determined well before the launching of a subsequent one so that weaknesses are not passed from one phase to the next.
    • APLs are appropriate structures for trying out and improving innovative systems and not for conventional programs of school building and materials distribution;
    • All project preparation should include the development of a project implementation plan and a careful analysis of implementing unit capacity, especially if -- like the Basic Education Project -- a project is relatively short and front-loaded; otherwise, long delays can be expected;
    • When Bank financing is in support of a largely Government funded program, its funding should focus on features for which it has a comparative advantage; if it funds the same components that the Government is funding, it may find its support being marginalized (for example, because of relatively difficult Bank procurement rules).
    • Projects must have clearly stated and measurable objectives, accompanied by outcome-based performance indicators, which should be used both in project supervision and evaluation. For education projects covering quality improvement, the outcomes indicators should include measures of student learning outcomes.
    • Concrete follow-up actions should be taken as the result of Quality at Entry and Quality Enhancement Reviews. Failure to do so means that Project design flaws, such as those found in the current Project, will not be adequately dealt with.
    • Realistic risk assessments (worst-case scenarios) should be made at project appraisal and contingencies designed to deal with them if they materialize.
    • Projects should include cost-effectiveness measures in their evaluation designs where there is heavy investment in infrastructure (e.g., buildings, computers, and IT facilities), where inflated unit costs are common. The Bank should also pay attention to cost effectiveness in its evaluation of such projects.

    8. Audit Recommended?  Yes

              Why?  This APL phase 1 is already being followed by phase 2, even though it was mostly unsatisfactory. It would be good to see whether the Government and Bank are making adjustments in the current project and institutional capacity on the basis of these findings; this is particularly crucial, since the funding levels of these APLs are high, and thus, the consequences of another failure serious. In addition, an OED country assistance evaluation for Turkey will be held in FY2006.

    9. Comments on Quality of ICR:

    The quality of the ICR is satisfactory overall; it does a good job of sorting out a complex and confusing project, gathering and objectively applying relevant evidence in its assessment, making reasonable judgments in its ratings, and generating a useful list of lessons learned. However, there are also some shortcomings, as follows:
  • The ICR could have broadened its analysis to components "not financed" by the Bank by considering Government figures and stakeholder input related to major project components such as Teacher Training, Recruitment and Deployment; Marmara Earthquake Relief; and Student Social Aid. For the two latter components, the Annex 2 expenditure tables actually show some Bank funds being disbursed.
  • The ICR's use of the PAD as the basis for ratings of project components, and not the amended Loan Agreement (which it used for evaluating objectives), meant that some added Project features were not evaluated, such as expansion of the Mother/Child preschool program. Also, the ICR could have highlighted the Project amendment scaling back the Monitoring Response Facility component to the development of a pilot program, with a drastically reduced budget (see amendment letters, Annex 1).
  • The ICR should have better explained the large differences between allocations to components and actual expenditures (e.g., the the almost 100% increase in actual expenditures for construction and upgrading of basic education schools over appraisal estimates). Also, the ICR gives two different figures for total Project costs ($2825 million on page 14 and $2515 million in Annex 2; the region subsequently clarified that the former is a typographical error).
  • Following the Intensive Learning Model, this ICR was expanded to include field visits and both a "Beneficiary Assessment," and a "Stakeholder Workshop." These two events were described at the end of the ICR but their results could have been more fully reflected in the main text of the ICR and its lessons learned.

  • © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions