Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Secondary Education Enhancement Project


  
1. Project Data:   
ICR Review Date Posted:
10/30/2013   
Country:
Egypt
PROJ ID:
P050484
Appraisal
Actual
Project Name:
Secondary Education Enhancement Project
Project Costs(US $M)
 250.0  320.2
L/C Number:
C3194
Loan/Credit (US $M)
 50.0  53.8
Sector Board:
Education
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  04/15/99
 
 
Closing Date
06/30/2006 10/31/2012
Sector(s):
Secondary education (80%), Tertiary education (5%), Central government administration (5%), Sub-national government administration (5%), Other social services (5%)
Theme(s):
Education for the knowledge economy (33% - P) Education for all (33% - P) Social analysis and monitoring (17% - S) Participation and civic engagement (17% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Susan Ann Caceres
Judyth L. Twigg Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:

According to the Project Appraisal Document (PAD, p. 2) the development objective was "to improve the quality and opportunity in secondary education by increasing access to general secondary education through upgrading commercial schools to technical schools and providing flexible options for study within and between branches of the system; better aligning curricula and assessment with the skills needs of employers and higher education; providing professional development for teachers and administrators on new technologies, curricula, assessment, and management techniques; and strengthening institutional capacity." Annex 1 of the PAD states the project development objective slightly differently: "to improve quality and opportunity of secondary education by increasing equality of opportunity in access to general secondary education through flexible options for study within and between branches of the system; better aligning curricula and assessment with the skills needs of employers and higher education and providing professional development for teachers on the new curricula and assessment techniques; and to strengthen the capacity of the education system at the central level and in selected governorates and schools to deliver quality education by better defining responsibilities, strengthening accountability mechanisms, and providing professional development for school heads and administrators."

The ICR (p. 3) notes that the statement of objectives in the PAD (p. 2) contained an error when stating "upgrading commercial schools to technical schools," since elsewhere in the document commercial schools are technical schools and the PAD clearly states there are too many technical schools. This review understands that the purpose was to upgrade commercial schools to general schools.

According to the Credit Agreement (Schedule 2), the objective of the project was "to establish the basis for an equitable education system, through quality and opportunity improvement and strengthening of management and accountability procedures."

The Guidance Note (paragraph 4 Resolution of Differences) given operational staff with respect to packages submitted to the Board states: “If there are differences of views between the lawyer, other reviewers, and/or the Task Team Leader as to the consistency of the project documents with the negotiated legal agreement(s), the negotiated legal agreements(s) prevail absent a reopening of negotiations to address the discrepancy.“ Thus, for this review, the objectives noted in the Credit Agreement will be used.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

There were two components:

Improving Quality and Opportunity (appraisal, US$232.6 million; actual, US$318.3 million) was to convert 315 commercial (or technical) schools to general schools to be able to reach a 50/50 split in the country between general and technical schools. During the first tranche, schools in 27 governorates were to be targeted for rehabilitation and upgrading to meet general education core course standards. Upgrades were to included elements such as technology, science laboratories, and resource centers to impart the new curriculum. School selection was to take into account factors such as rural/urban enrollment in each governorate and the respective enrollment of commercial students. A framework for a comprehensive general curriculum was to be developed. Teachers were to receive training to learn how to integrate technology into classroom practices and the core curriculum.

Strengthening Institutional Capacity (appraisal, US$17.4 million; actual, US$1.9 million) envisaged several activities to improve local school management and community and business involvement, as well as develop management reform at the central level. It was to support community and private sector involvement for local management through the use of school councils. School councils were to receive training in their new roles and responsibilities and were to be provided small grants to be implemented at the school level with supervision at the muddiriya level. Grants were to be disbursed at the local level and audited by the muddiriya and national levels. This component was also intended to develop new quality assurance mechanisms and improve management practice, building school leadership capacity to better plan and coordinate with school councils, as well as other instructional leadership responsibilities.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Cost: Total project costs were US$320.2 million (128% higher than estimated). Component one was US$318.3 million, which was 179% higher than planned. Spending on component two was much lower than planned at US$1.9 million, or 12% of the planned amount. The ICR does not explain why spending was so different than originally planned.

Financing: An IDA Credit in the amount of US$ 50 million financed the project.

Borrower Contribution: The Borrower contributed US$ 242 million, which was 121% higher than originally planned. The ICR does not state the reasons for the higher amount.

Dates: The project closed six and a half years later than estimated. The ICR does not adequately explain the reasons for the extensions. On March 21, 2006, the project's closing date was changed from June 30, 2006 to June 30, 2008, and funds were reallocated from the school fund to school upgrading. On June 29, 2008, the closing date was extended from June 30, 2008 to June 30, 2010. On June 29, 2010, there was a third extension of the closing date from June 30, 2010 to June 30, 2012. The school conversion program was suspended due to public resistance, and so the remaining funds were reallocated to upgrading schools, as well as the introduction of minor changes to the indicators. On July 3, 2012, the closing date was extended for the fourth time from June 30, 2012 to October 31, 2012.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Relevance of Objectives is rated Substantial.

At appraisal, there were several key issues related to secondary education. The curricula and teacher quality were low. There were inadequate quality assurance mechanisms to monitor school performance and student learning. Management of the education sector was highly centralized and inefficient in its use of resources. Government policies limited entry of students to general secondary education and oriented most (70%) toward vocational education. This vocational bias persisted until the time of project preparation (late 1990s). At that time, the Government developed a Strategic Framework for Reform of Secondary Education, which sought to make secondary education more equitable and responsive to labor market needs by improving quality and strengthening institutional capacity. The objectives of improved quality, more equitable opportunities, and strengthening management and accountability of secondary education were therefore substantially relevant to Government priorities and to the Bank's most recent Country Assistance Strategy (2006-20), which focuses on higher quality of education and increased employability of graduates (p. 37). Although the Bank's current Interim Strategy Note (2012) describes the Bank's focus in education primarily on increasing early childhood education in rural and poor urban areas (p. 18), it also notes activities related to pre-university education and a continued focus on strengthening monitoring and planning in education (p. 20).

b. Relevance of Design:

Relevance of Design is rated Modest.

There was a logical connection between the project's objectives, planned activities, and measures. However, the Results Framework was overly ambitious, particularly considering that no baseline data were collected during preparation.

Design did not employ alternative activities that could have achieved the stated objectives with equal or greater efficacy and/or efficiency, including: using available places in existing general secondary schools, improving commercial schools by updating their curricula and improving their facilities and staffing, reducing entry requirements into general education, or supporting the Government's own ongoing school building program. The ICR (p. 19) points out a number of additional weaknesses in project design: over-ambitiousness of the scope of intended educational change, and lack of an adequate monitoring and evaluation framework. Details were also lacking in some of the planned activities, which needed to be further defined during implementation. For example, the specific activities related to quality assurance mechanisms and management practices at the central levels were not developed, listing instead potential technical assistance activities.


4. Achievement of Objectives (Efficacy) :

The objective noted in the Credit Agreement to "establish the basis for an equitable education system, through quality and opportunity improvement and strengthening of management and accountability procedures" has, for the purposes of this Review, been broken into three sub-objectives: Quality Improvement, Opportunity Improvement (access), and Strengthening Management and Accountability. The evidence of attainment of each is noted below.

Overall, the ICR (p. 21) states that "even with four closing-date extensions and 13 years of project implementation, only two of the 18 targets were achieved. For most of them, there was no record of progress or effort of any kind."

Quality Improvement: Modest

Outputs:

A core curriculum framework was developed and approved. No curriculum framework existed at the beginning of the project. This framework included a unified 10th grade core curriculum for technical and general secondary schools, core and elective curricula for general and technical education, and other materials. The curriculum was consistent with international best practice. The Ministry issued a decree authorizing private publishers to develop new textbooks under the new curriculum framework for competitive selection. However, little progress was made in aligning assessment methods and instruments with the framework. The framework has not been implemented, but this can be attributed to disruptions surrounding the country's 2011 revolution and aftermath (ICR, p. 25).

593 existing general secondary schools were upgraded with facilities and equipment such as computer labs, science labs, multimedia rooms, furniture, and books for school libraries. These inputs could be expected to improve the conditions of learning, if there is significant use of those upgraded facilities. However, a recent study suggests that the association between the upgraded facilities under the project and student achievement has been modest (ICR, p. 29), which may be related to reported under-utilization of science labs.

30% of project schools complied with national accreditation standards for facilities and learning equipment, which met the target (30%). However, a 2006 study raised questions about whether school upgrading could lead to improved student learning, as there was no consistent relationship between school resources and student achievement. This study also found that most students selected humanities specializations rather than science specializations, suggesting under-utilization of laboratories.

86,000 teachers were trained in the use of upgraded facilities and equipment. There is no evidence provided on the use and effectiveness of this training.

48,000 teachers were trained in more effective teaching methods. The ICR (p. 51) reports that the training covered topics such as active learning, cooperative learning, constructive learning, evaluation, and use of technology. There was no evaluation of the effectiveness of the training. The ICR reports that there was a high level of participant satisfaction with the training (ICR, p. 25). There is no evidence provided on the use and effectiveness of the training.
Staff in the National Examination Center were provided training in assessment methodologies. Egypt participated in the 2007 Trends in International Mathematics and Science Study (TIMSS) student assessment in science and math.

Outcomes:

Data were not available (since not tracked) on the share of technical secondary graduates entering higher education, improved employability and/or grades of graduates, or other quality indicators.

Opportunity Improvement (access): Modest

Outputs:

205 commercial schools were converted into general secondary schools, which did not meet the target of 315. This represented 28% of the 722 commercial schools in operation between 2001-2004. School conversion was suspended by the Government in 2004, given negative public sentiment. Instead, the project shifted its focus to upgrading of existing general secondary schools. 593 existing general secondary schools were upgraded. This changed the beneficiary focus from students in commercial schools (who face limited options upon graduation) to those already in general secondary schools. This shift in activity was not consistent with the project's development objectives (ICR, p. 10) and meant that the original goal of reaching a 50/50 balance of commercial and general secondary schools was not reached.

There was limited progress to improve flexibility in the secondary education system to allow students to transfer between general and vocational education, or on the creation of additional electives (ICR, p. 24).

Outcomes:

The share of general secondary students increased from 30% of all secondary students in 1999 to 43% in 2012, which met the revised target of 40%. Other Ministry of Education data, however, report a lower figure than 43%, and it is not clear that this achievement can be attributed to project-financed activity. Simultaneously with the project, the Government raised the number of student placed in general schools and reduced the number of student places in commercial/technical schools, and it also continued an ongoing school building program. According to the ICR (p. 24), attributing an appropriate share of the credit to each of these sources would require a more refined analysis that takes into account a large number of factors outside the purview of the project.

It is difficult to compare secondary enrollment and student performance in the years 2008 through 2010 to that of prior years, given a new Government policy in 2005 of extending primary education to six years. Enrollments remained below normal for the next five years and then stabilized in 2011.

Strengthening Management and Accountability: Modest

Outputs:

70,000 secondary school principals and deputy principals were trained in the use of technology for school management. The impact of this training was not assessed (ICR, p. 23).

1,354 Ministry staff (quality and monitoring reviewers) were trained in quality assurance and accreditation (ICR, p. 53). This training became the basis for establishing a quality assurance department in each idara across the country.

Criteria for teacher promotion and selection of principals were developed.

The project provided computers and computer labs to a number of educational institutions to strengthen their capacity (ICR, p. 54).

1,000 local School Boards of Trustees were trained. This training addressed roles, school functions, and monitoring of school quality. All project schools now have a Board of Trustees. While schools prepared action plans, these were not sustained systematically in schools (ICR, p. iv). The percentage of school management teams judged competent in their duties was not tracked. The ICR points out that no baseline measure was available at the time of preparation, and the end value is based on information about training, as no performance measurement criteria were developed (p. iii).

Planned activities related to new job descriptions for positions within the inspectorate and development of a quality assurance mechanism were not accomplished (ICR, p. 42).

Outcomes:

None reported.


5. Efficiency:

The PAD (pp. 14-17) calculated an internal rate of return of 23%, based on the assumption that the quality enhancements in secondary education would provide benefits that are a significant percentage of secondary graduates' wages.

According to the ICR (p. 43), the PAD's cost-benefit analysis was based on unsupported assumptions about earnings of graduates from upgraded schools and how enrollments might respond to changes. Also, the PAD presented no comparison of the actual earnings of general secondary and technical secondary graduates in the labor market and their respective earning differentials. Given these shortcomings, the ICR did not update the analysis. Furthermore, the original assumptions may have reversed, as a recent study in Egypt found the returns to general education were lower than that of technical education in 2006 (ICR, p. 27). The delay in project implementation could also have lower the internal rate of return.

The overall goal of increasing the share of general secondary enrollment, given the 2006 finding of lower returns to general education, raises serious concerns about the project's efficiency, given that this aspect (school conversion) accounted for US$199 million of the US$250 million planned project costs (ICR, p. 20). The ICR also suggests that other options, such as lowering the admission threshold at entry into existing general secondary schools with excess capacity, or giving priority to building new general secondary schools under the Government's ongoing program, could have more efficiently achieved the project's intended purpose.

Efficiency is rated Modest.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
23%
100%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Relevance of objectives is substantial, while relevance of design, achievement of all objectives, and efficiency is modest. The project implemented activities to improve quality by upgrading school facilities, training teachers, school committees, and principals, and developing a new curriculum framework. Due to design and implementation weakness, however, there were shortcomings related to the appropriateness and implementation of the school conversion activities. No evidence is presented on outcomes related to quality improvement. Also, there was no progress implementing the new curriculum framework and building more flexibility between technical and general education.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

There are several risks to the development outcome, which combined pose a high risk. First, the Government lacks a plan for achieving the key educational outcomes that were not attained under the project. Second, it is unclear whether Government priorities will change, giving the public resistance experienced in this project and the public sensitivity that could emerge with the implementation of the new curriculum. This is compounded by differing secular and religious interests, which could affect some of the education reforms supported by the project. Third, private tutors resist implementation of the new curriculum, as they want the status quo to remain and students to seek their services during the final year of secondary schooling. Fourth, there remains high overall political instability.

a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:

Project preparation benefited from the experiences of the Bank and other donors in working in the education sector. While other donors were active in education in Egypt, they were focused on levels other than secondary. However, there were several shortcomings in preparation that negatively impacted implementation.
Preparation left a number of options related to the curriculum framework, assessment, instructional materials, and training to be later developed and/or decided. There was inadequate examination of alternatives to reach the project's aim of increasing the proportion of students in general education. There was insufficient analysis to show the difference in labor market outcomes between general secondary and commercial secondary schools, which would have provided a basis for making informed decisions. Also, the school grants program was not adequately designed; appropriate planning would have detected the fact that funds were not allowed to be disbursed directly to schools due to difficulties in tracking; the grants program was cancelled in 2006. Insufficient social analysis and public outreach was done at entry. More work in these areas may have detected public uneasiness with school conversion, particularly the resistance from parents with girls. There was an overly optimistic assessment of the Government's ability to enact education reform, and the scope and pace were unrealistic. Given low M&E capacity, the project's M&E plans were ambitious, particularly considering that no baseline data were collected during preparation.

Quality-at-Entry Rating: Unsatisfactory

b. Quality of supervision:

Staffing discontinuities in the Bank hampered adequate supervision, particularly during the initial years. Supervision efforts before 2007 reflected insufficient attention to project outcomes. There was much turnover, until the final years (beginning in 2007) when continuity at the TTL position was maintained. Improved project supervision was evident with the final TTL and the new Sector Manager and Country Director, as they tried to address implementation problems and provide corrective actions. During this time period, the Bank hired an education consultant to help prepare a new curriculum framework and organize an evaluation of the training provided under the project.

The supervision team did not effectively use ratings as a signal to the Borrower and Implementing Agency. Satisfactory and moderately satisfactory ratings were given for development outcome and implementation progress up until 2009. Also, despite lack of progress in M&E, it was not until November 2004 that M&E performance was rated as unsatisfactory, and even then no rationale for downgrading or suggested corrective actions were provided. Despite the Government not completing necessary covenants related to M&E (hire a full-time expert on M&E and establish an M&E plan), the Bank granted the first extension to the closing date. The ICR notes shortcomings in the Bank's internal documents and reporting (changes were not fully documented, such as deletion of school grant activity or suspending conversion of schools, no Aide Memoire for the Mid-Term Review mission, inconsistencies in the wording of the development objectives between the PAD and Credit Agreement) (ICR, pp. 11, 10, 16). Bank management was not properly consulted regarding the decision to suspend the conversion of commercial schools and proceed with general school upgrading.

Given that monitoring and evaluation was not sufficiently emphasized by the Implementing Agency, there was a need for the Bank to provide more supervision, yet attention was lacking during Bank supervision missions in the first four years of project implementation (13 Implementation Status Reports between May 1999 and May 2004 reported the same issues and actions for M&E). The Mid-Term Review did not make recommendations to strengthen M&E, despite noted shortcomings. The Mid-Term Review left undiscussed several important issues (suspension of commercial school conversion and replacement with general school upgrading). The Mid-Term Review occurred in March 2005, fifteen months before the initial closing date, which diminished the ability to make early corrections.

Quality of Supervision Rating: Unsatisfactory

Overall Bank Performance Rating: Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

In 2011 there was widespread civic and political disruption, which contributed to delays in project implementation and in the release of official enrollment statistics.

There were a number of shortcomings in performance that were under the control of the Government. Frequent changes in Ministers of Education (six between 2004 and 2012) led to delays and discontinuity. There were long delays in the Government's release of funds. The Government did not prepare a plan for maintenance, despite requests from the Bank. There was failure to comply with covenants for granting extensions. A full-time project manager was not appointed until 2005, six years after approval. Other changes were made that created additional bureaucracy for project implementation.

Government Performance Rating: Unsatisfactory

b. Implementing Agency Performance:

Implementation responsibility rested with a Project Planning and Monitoring Unit (PPMU) until March 2010, when the Government shifted implementation management responsibilities to the Education Support Fund. During the final two and half years of implementation, there were solid efforts to complete school upgrading and training activities. However, this could not make up for the lack of commitment during most of the project. Implementation was negatively impacted by staffing discontinuities. It took considerable time for a M&E specialist to be hired and a plan developed. Monitoring and evaluation staff did not have technical competencies in these areas. Up to the time of the Mid-Term Review, implementation progress was largely limited to hardware procurement related to school conversion, and there was limited progress on the quality improvement activities. Implementation efforts suggest insufficient attention to project outcomes. Financial management issues were encountered (see Section 11b).

Implementing Agency Performance Rating: Unsatisfactory

Overall Borrower Performance Rating: Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

Tracer studies and employer surveys were part of the M&E design to track project outcomes. A tracer study was to be carried out during project preparation to provide baseline data, which was not done at entry or subsequently. The PAD presented plans for outcome evaluations such as employability of graduates and assessments of the training provided by the project.

b. M&E Implementation:

Baseline surveys were not carried out. Indicators were changed during implementation, as there was a suspension of converting commercial schools to general secondary schools. The targets for indicators were revised. New indicators were also added, containing performance measures or benchmarks to assess attainment. These new indicators required the creation of additional instruments (minium standards for school infrastructure and learning equipment, mastery of curriculum framework) that were not developed. The only monitoring and evaluation carried out under the project consisted of monitoring outputs. There was no proper assessment and documentation of the attainment of all indicators (ICR, pp. 39-42). A stakeholder perception workshop was held in the final months of the project. There was no evaluation of project outcomes (such as new concepts and skills learned, changes in teaching practices, learning outcomes for students, or labor market outcomes), as planned in the appraisal document.

a. M&E Utilization:

There was low utilization of data, as there was inadequate attention given to monitoring and evaluation during implementation.

M&E Quality Rating: Negligible

11. Other Issues:

a. Safeguards:

In the appraisal document, the project was classified C for Environmental Assessment (OP 4.01), as construction was limited to existing schools. The ICR does not describe the project's compliance with this safeguard, nor Bank supervision of the safeguard.

b. Fiduciary Compliance:

There were issues with financial management performance, as ratings were downgraded to unsatisfactory, but by 2011 these issues had been resolved. The implementation of recommended improvements in financial management is still pending. The ICR does not provide information about procurement, and it provides little detail on the issues encountered with financial management. The project team later reported that procurement was adequate, as the issues raised by the Bank were addressed timely and satisfactorily. The project team also stated that financial management reports were submitted to the Bank on time. Issues in financial management occurred when project coordination responsibilities were transferred toward the end of the project; there is still no detail provided on the substance of these financial management issues.

c. Unintended Impacts (positive or negative):

The school conversion component was designed to allow students who would have gone to technical schools the option to attend general secondary schools. Students who attend technical schools are often from poorer families. Changing the project's activities from technical school conversion to upgrading of general secondary schools may have had the unintended consequence of benefiting students from higher-income families, reducing the project's intended equity benefit. However, many of the schools selected for upgrading were located in low-income areas.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Risk to Development Outcome:
High
High
 
Bank Performance:
Moderately Unsatisfactory
Unsatisfactory
There were significant shortcomings in Quality at Entry (see Section 8a above). Based on the Harmonization Criteria, the outcome rating determines the overall performance rating when there is a split between the ratings for Quality at Entry and Quality of Supervision. 
Borrower Performance:
Moderately Unsatisfactory
Unsatisfactory
There were significant shortcomings in both Government and Implementing Agency Performance.  
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

IEG has synthesized the lessons presented in the ICR (pp. 35-37) to the following:
  • Proper design and evaluation, as well as realistic objectives, are required. Given that this was the first Bank project in secondary education in Egypt, the goals and objectives needed to be modest, not attempting to address all issues. In this project, there was insufficient attention to analyses to develop the justification for the school conversion component, which led to design weaknesses. Preparation neglected baseline data collection, which was compounded by the lack of M&E expertise and inadequate attention to evaluation during implementation.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR provides a candid assessment of implementation weaknesses. It describes implementation thoroughly and discusses attainment of the project's objectives. The analysis tries to go beyond a presentation of indicators and present other sources of data. The Borrower's comments were helpful and were used to supplement this Review. However, it is a serious shortcoming that the ICR does not discuss environmental safeguard compliance or procurement management, and provides limited information on financial management. The ICR is also missing important information on the reasons underlying the difference between appraised and actual costs by component, and on the reasons for the long and multiple extensions of the project's closing date.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions