Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Education Quality Improvement Program (eqip) Phase 2


  
1. Project Data:   
ICR Review Date Posted:
01/10/2012   
PROJ ID:
P082999
Appraisal
Actual
Project Name:
Education Quality Improvement Program (eqip) Phase 2
Project Costs(US $M)
 290.92  260.0
Country:
Tunisia
Loan/Credit (US $M)
 130.34  129.72
Sector, Major Sect.:
ED: Central government administration, Sub-national government administration, Primary education, Secondary education
Cofinancing (US $M)
 0  0
Theme(s):
Education for the knowledge economy (50% - P) Education for all (50% - P)      
L/C Numbers:
L7220
Board Approval (FY)
  2004
Partners involved
 
Closing Date
06/30/2008 09/30/2010
         
Evaluator: Panel Reviewer: ICR Review Coordinator: Division:  
Judith Hahn Gaubatz
George T. K. Pitman IEG ICR Review 1 IEGPS1

2. Project Objectives and Components:

a. Objectives:
The project was the second part of a two-phase Adaptable Program Loan (APL, 2001-2007), whose overall purpose (according to the Project Appraisal Document (PAD, page 3) was to support the Government's objectives to achieve near-universal completion of basic education, provide a greater number of students (from a wider range of backgrounds) with opportunities for post-basic education, and modernize the sector to improve the quality and efficiency of outputs.

According to the PAD (page 3), the objectives of this second-phase project were (i) to promote excellence in teaching and learning throughout the school sector, while (ii) continuing the push for equity by ensuring the inclusion of all children at all levels of the basic education system.

According to the Loan Agreement (LA, page 13), the objective of the project was to reinforce the process of reforming the education sector through (i) promoting excellence in teaching and learning in an improved pedagogical and physical environment; and (ii) ensuring that all children are provided with equitable and adequate opportunities to access quality education at all levels of the school system.

As the LA statement of objectives is more specific, it is used as the basis for this review.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components (or Key Conditions in the case if Adjustment Loans):
(Note: The actual project costs by component were not available. According to the project team, these figures were not tracked.)

1. Inclusive Basic Education (Appraisal: US$32.1 million; Actual: not available): This component aimed to improve the quality of education through introducing a new skills-based teaching methodology (Approche par competence - APC), modernizing the teaching of languages and science, and introducing an optional studies program. It also aimed to integrate and mainstream vulnerable children into regular schools and to enhance the quality of teaching in low-performing schools. Activities included developing new curricula and teaching materials, developing a learning assessment system, expanding the Priority Schools Program (PEPE) which addressed the needs of low-performing schools, implementing a program to integrated disabled children, and providing equipment and training,

2. Diversity in Secondary Education (Appraisal: US$5.0 million; Actual: not available): This component aimed to expand the options in secondary education to meet the needs of an increasingly diverse student population and to provide access to information regarding career paths and education programs. Activities included developing curricula and teaching materials for technology programs, evaluating the use of science and technology materials, establishing an Information and Guidance Center, and providing teacher training.

3. School Sector Management (Appraisal: US$5.2 million; Actual: not available): This component aimed to decentralize and modernize the school management system. Activities included establishing school improvement programs, developing a management information and communication technology (ICT) system, and providing training for users.

4. Strengthening Core Systems (Appraisal: US$218.2 million; Actual: not available): This component aimed to develop a strategy to restructure teacher training, integrate ICT in the learning experience, and ensure regular assessment of learning achievement. Activities included developing an in-service teacher training system, a communications strategy on the main themes of school reform, and ICT guidelines for teachers and schools.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project cost and financing:

  • Additional resources became available due to the appreciation in the exchange rate. It was agreed to use the extra resources to increase the number of schools to be rehabilitated.
  • Funds were reallocated on two occasions to support scaling up of the ICT program and the equipping of language labs. As the reallocations involved more than five percent of project costs, Project Papers were prepared and subsequently approved by the Regional Vice-President (RVP).
  • The ICR does not provide actual project costs by component, only loan costs..

Borrower contribution:
  • The Borrower provided US$82.8 million out of the planned US$160.6 million, or 51.6%. The ICR does not provide a reason for the shortfall.

Project dates:
  • The closing date was extended from June 2008 to June 2009 to allow for the completion of activities which had experienced implementation delays, as well as those that were added due to the favorable exchange rate.
  • The closing date was extended a second time from June 2009 to June 2010 to allow for the completion of several large procurements, which had experienced delays.


3. Relevance of Objectives & Design:

Objectives: High. The project objectives were highly relevant to the government's education reform priorities, including improving learning outcomes for vulnerable children, increasing the diversity of secondary education opportunities, and adapting the school system to the needs of the economy. The two-phase APL was intended as a key instrument for implementation of this reform agenda, and the continuing focus on these priorities in yet another follow-up Bank project further points to their high level of relevance. The Country Partnership Strategy for FY2010-2013 emphasizes results in the policy areas of improving employability and achieving the Millennium Development Goals (MDGs) in Education (increasing the primary completion rate and improving the quality of education).
Design: Modest. The project aimed to further the achievements of the first phase of the APL program, which focused on improving the quality of basic education, by improving teaching methodology, ensuring access, and diversifying the curriculum. The choice of an APL was appropriate given the government's fiscal constraints balanced against the need for major investment at the secondary education level.

However, while project activities were likely to contribute to the ultimate outcomes of increased enrollment and completion rates, intermediate outcomes that were more directly linked to the project's outputs should have been identified. In addition, the institutional arrangements were complex and called for "a wide range of actors and institutions within the education system to work in a collaborative and coordinated way which was not, and is still not, a practice observed in the Tunisian context" (ICR, page 11).

4. Achievement of Objectives (Efficacy) :

Objective: To Reinforce the Process of Reforming the Education Sector through promotion of excellence in teaching and learning in an improved pedagogical and physical environment and, ensuring all children are provided with equitable and adequate opportunities to access quality education. Substantial


The education reform agenda included the following: extending improvements of learning outcomes to the most vulnerable children; responding to the increasing diversity of the secondary education population by widening the range of opportunities; enabling the secondary school system to adapt to the needs of the economy; and taking advantage of the "lull" in the size of the lower basic education population to focus resources in cost-effective ways.

The project objective was to be achieved through the following two means:

Promoting Excellence in Teaching and Learning in an Improved Pedagogical and Physical Environment

Outputs

  • 1,000 trainers, 800 principals, and 42,542 teachers were trained on APC methodology. No targets or coverage levels provided.
  • Creation of an Information and Guidance Center to facilitate career counseling and guidance services. The Center provided a database on alternative and available options regarding university study opportunities and technical and vocational training. 1,016 teachers received training on using the Center's service.
  • Renovation of over 1,100 schools and 200 classrooms to increase infrastructure capacity (i.e. science and technology classrooms, laboratories, and boarding facilities). No target provided.
  • Operationalization of education management ICT system in all schools, including training of 2,236 users and 1,049 technical staff. No target provided.

However,
  • The school improvement and school council programs were not, according to the ICR (page 32), implemented as widely as planned (the ICR does not indicate the extent to which these activities were implemented) , This was in part due to lack of support from teacher unions, as well as the political and cultural context in the country which made promotion of school autonomy difficult as an overall education policy goal.
  • A communication strategy covering the main themes of school reform was not carried out.

Outcomes
Note: The baseline year reported is 2003/04, and final year is 2009/10.

Completion rates
  • The completion rate for boys in basic education increased from 42.5% to 51.0%, surpassing the target of 48.0%.
  • The completion rate for girls in basic education increased from 60.1% to 69.0%, falling slightly short of the target of 69.5%.
  • The completion rate for boys in secondary education increased from 42.9% to 45.3%, falling short of the target of 54.3%. The ICR (page 18) notes that changes in the baccalaureate exam may have accounted for the changes in the indicator and that there remained "high repetition and drop-out rates among boys" despite progress in terms of absolute numbers and percentage of completers.
  • The completion rate for girls in secondary education increased from 45.5% to 60.6%, falling short of the target of 67.9%. See above comment on the effect of changes in the baccalaureateexam.

Learning outcomes
  • Improved scores in 2007 TIMSS, 2006 PISA, and 2009 PISA international student assessment exams. While the actual data is not cited in the ICR, the project team noted that the data was published by OECD.

Other
  • The share of non-salary expenditures at first stage of basic education increased from 2.5% to 3.2%, surpassing the target of 2.5%.
  • The share of non-salary expenditures at second stage of basic education and secondary education decreased from 6.4% to 4.5%, falling short of the target of 6.4%. However, unit costs for secondary education increased in real terms by 57%, which the ICR (p. 19) suggests may be a reflection of the policy goal to invest in improving quality. The ICR also (page 18) reports that there was a significant change in the student/teacher ratio, which went from 20.4 to 13.4, resulting in an increase in the salary mass and therefore its relative share in the budget. It also notes that the results of this indicator may also have been due to the prevailing practice of categorizing as "investment" the expenditures on ICT equipment and some teaching and learning materials.
  • The ratio of students per computer at first stage of basic education improved from 129 to 23, surpassing the target of 36.
  • The ratio of students per computer at second stage of basic education improved from 99 to 23, surpassing the target of 69.
  • The ratio of students per computer at secondary education improved from 45 to 19, falling short of the target of 15.

Ensuring that All Children are Provided with Equitable and Adequate Opportunities to Access Quality Education, which included increased access in low-income areas and for children with special needs.

Outputs
  • Expansion of the PEPE school program to 557 primary schools and 101 lower secondary schools, and expanding the number of beneficiary students from 7,667 in 2001 to 157,806 students at project closing. No targets provided. The program provided funding for school improvements, training for school staff, increased school hours, social support to children with special needs, generalization of canteens, and monitoring and evaluation by the coordination unit of the PEPE program.
  • Rehabilitation and equipping of kindergartens serving 1450 children from low-income areas. No targets provided.
  • Completion of program for integration of children with disabilities in 336 schools, covering 1219 children. No targets provided. The program included extensive training for teachers and school leaders, as well as dissemination of guidelines on integrating children with disabilities and rehabilitation of facilities to enable access.

Outcomes
Note: The baseline year reported is 2003/04, and final year is 2009/10.

Enrollment rates
  • The enrollment rate for boys aged 6-11 years old increased from 96.8% to 97.9%, surpassing the target of 97.2%.
  • The enrollment rate for girls aged 6-11 years old increased from 97.0% to 98.5%, surpassing the target of 97.5%.
  • The enrollment rate for boys aged 6-16 years old increased from 90.0% to 91.3%, surpassing the target of 90.6%
  • The enrollment rate for girls aged 6-16 years old increased from 91.0% to 92.8%, surpassing the target of 91.6%.
  • The enrollment rate for boys aged 12-18 years old increased from 73.5% to 74.7%, surpassing the target of 74.4%.
  • The enrollment rate for girls aged 12-18 years old increased from 77.5% to 81.4%, surpassing the target of 78.4%.

5. Efficiency (not applicable to DPLs):

Efficiency: Modest, due to lack of information in the ICR on cost-effectiveness and the efficient use of project resources. In addition, project implementation experienced considerable delays.

The PAD included a fiscal analysis of government spending on primary vs. secondary education because the demographic trends indicated that the population eligible for primary education was expected to decline, and expected to increase for secondary education. Fiscal efficiency was not expected to improve significantly, as the demographic changes were viewed as an opportunity to invest in improving quality, rather than to simply decrease costs. There was no economic analysis conducted.

Over the course of the project period, more schools were rehabilitated than originally planned; however, the ICR reports (para. 35) that "it became clear that additional resources in Tunisian Dinars would be available due to appreciation in the exchange rate of the Euro against the local currency. It was thus agreed to use the extra resources to increase the number of buildings in primary and secondary schools to be rehabilitated." This indicates that the increase in number of schools was more due to the injection of additional funds from favorable changes in the exchange rate, rather than efficiencies in project implementation.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:



Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The project objectives were highly relevant, although the project design was modestly relevant due to complexity in the institutional arrangements. The objective of promoting excellence in teaching and learning and ensuring equitable access was substantially achieved, as reflected by gains in enrollment and learning outcomes, though attribution to the project somewhat tenuous. Efficiency was rated modest due to lack of information on efficient use of project resources specifically and implementation delays.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating :

The ICR reports (page 16) that during the last year of implementation, the Bank and government engaged in a dialogue to analyze lessons learned in four critical areas: use of ICT, integration of children with disabilities, use of school improvement plans, and school management decentralization. Government officials prepared analytical reports on each of the topics and the lessons are being integrated into a new education project that is under discussion.


The risk of resistance from certain stakeholders (teacher unions) was assessed in the PAD as moderate; however, mitigation measures were not effectively implemented and the risk subsequently materialized and led to incomplete implementation of some activities.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

Quality at entry: The project was prepared following successful completion of the first-phase of the APL program and incorporated experience from that project. However, the ICR notes (page 11) that a more analytical and objective assessment of weaknesses could have benefited the subsequent project design. The ICR (page 18) also reports that "both the Government and the Bank probably underestimated the resistance and reluctance of teacher unions, principals, and regional administrators towards school-based improvement plans, curriculum development, and management." Shortcomings in the M&E framework were significant (see section 10).

Supervision: The ICR reports (page 23) that supervision visits were regular, well organized and adequately staffed, with aide-memoires and implementation supervision reports (ISRs) informative and concise. The Bank team repeatedly raised issues in supervision missions such as the M&E framework and procurement staffing, although not all issues received an adequate response from the government.

a. Ensuring Quality-at-Entry: Moderately Unsatisfactory

b. Quality of Supervision: Satisfactory

c. Overall Bank Performance: Moderately Satisfactory

9. Assessment of Borrower Performance:

Government: The Government demonstrated a high level of commitment, as reflected by the short time period between project approval and effectiveness (two months), and provided an overall supportive policy environment, due to the close alignment with the government's own education reform agenda. In addition, cumbersome procurement processes and the failure to appoint a project coordinator for the initial project period led to significant implementation delays.

Implementing Agency: The various implementing departments carried out most of their planned activities; however, the ICR notes (page 13-14) that there was no project coordination for a period of 18 months, and that there was inadequate overall coordination leading to a lack of "a comprehensive vision of the project, i.e. how specific interventions were to contribute to improving quality and equity in education." Procurement delays were significant, although the situation improved after the assignment of staff to the specific types of procurement (construction, rehabilitation, equipment and selection of consultants), and there were some issues of lack of full compliance (See Section 11). The ICR reports (page 14) that the government was "adamant about keeping the original results framework, because of its consistency and alignment with its own education reform strategy." The inadequacy of the results framework persisted, despite the Bank team's numerous proposals to add more specific and project-related indicators.

a. Government Performance: Moderately Unsatisfactory

b. Implementing Agency Performance: Moderately Unsatisfactory

c. Overall Borrower Performance: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

Design: While the M&E framework was consistent with the government's M&E priorities, the choice of indicators was inadequate for the purposes of assessing the achievements of the project specifically. As noted in the ICR (page 6), "all selected indicators are broad system-level ones, which makes it difficult to claim any kind of attribution of project interventions to the accomplishment - or lack thereof - of the target values." Indicators that more directly measured the outcomes from specific project activities were not identified.

Implementation: The M&E arrangements in the original project design, albeit with the noted shortcomings, were carried out as planned, including evaluative studies of some of the key project interventions. The grade 4 national assessment was conducted in 2004 and included in the overall M&E system. As noted previously, the results framework remained unchanged during the project period despite the repeated proposals from the Bank.

Utilization: Although data from the M&E system were used to track implementation progress, the ICR notes (page 11) that "the project design did not sufficiently emphasize the need to use results monitoring to provide meaningful feedback to decision makers in the Government." The results of the grade 4 assessment were not made available by the government and thus could not be used for project M&E purposes. The ICR does note that the observation of continuous and invariable high repetition and drop-out rates triggered the decision to commission a study to analyze their underlying causes.

a. M&E Quality Rating: Modest

11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts):


Safeguards: The project was categorized as a "B" project under O.P. 4.01 Environmental Assessment. The project required an environmental assessment, which had been carried out under the first phase project. The ICR reports (page 16) that the Bank only financed schools approved in accordance with the environmental assessment guidelines. No major issues with regards to land acquisition were noted. The ICR reports (page 16) that the results of applying the land acquisition assessment procedure in the first phase project showed that private owners in fact benefited from donating land for school construction and that for each site acquired by donation, a land acquisition assessment was conducted prior to construction to ensure that it would cause no negative impact on persons or groups, nor create issues related to OP/BP 4.12.

Financial management: Overall, the project's financial management staff demonstrated adequate capacity and no significant financial difficulties were noted. Annual audits revealed no significant accounting issues. However, there were delays in producing reports due to the number of transactions involved and the number of entities in the regional level that resulted in coordination problems, flow of information bottlenecks, and reporting delays.

Procurement: Procurement experienced severe delays on a number of occasions, due to the absence of a project coordinator as well as the "rather cumbersome procurement practices" in the country. These delays contributed to two closing date extensions. There were also compliance issues with regard to the selection of consultants, which the ICR reports (page 25) was often due to "pressure exerted by the tender board chairs (who report to the Prime Minister and not to the MET) to apply inadequate national procedures." The ICR does not report on whether these compliance issues were satisfactorily resolved.


12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
Efficiency was rated Modest due to lack of information on efficient use of project resources. 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 

13. Lessons:

  • Advocacy and communication efforts should be directed towards groups that may be politically opposed to certain reform goals, such as teacher unions and school administrators.
  • Linkages of M&E to policy development (i.e. demonstrating improved quality of education) should be made clear to all stakeholders to ensure an adequate and effective M&E framework.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR provides an adequate picture of project implementation, despite the shortcomings in the M&E framework. Reporting of output targets (i.e. number of PEPE schools to be supported, number of secondary schools to receive infrastructure upgrades), if available, would have been helpful to better establish the extent of project achievement. The ICR notes (page 27) that at the time of the ICR draft review meeting (March 15, 2011), despite several reminders by the Bank team, the Borrower had only sent minor edits on the draft ICR and did not provide extended comments or feedback.

a. Quality of ICR Rating: Satisfactory

(ES-Rev5-Jul/06)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions