Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Wbg: Tertiary Education Project


  
1. Project Data:   
ICR Review Date Posted:
05/02/2014   
Country:
West Bank & Gaza
PROJ ID:
P083767
Appraisal
Actual
Project Name:
Wbg: Tertiary Education Project
Project Costs(US $M)
 15.9  21.6
L/C Number:
Loan/Credit (US $M)
 10.0  15.0
Sector Board:
Education
Cofinancing (US $M)
 5.9  6.6
Cofinanciers:
European Commission
Board Approval Date
  04/26/2005
 
 
Closing Date
12/31/2010 12/31/2012
Sector(s):
Tertiary education (80%), Central government administration (20%)
Theme(s):
Education for the knowledge economy (67% - P) Administrative and civil service reform (33% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Erik A. Bloom
Judyth L. Twigg Lourdes N. Pagaran IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Trust Fund Grant Agreement (pages 11-12), "the objectives of the Project are to assist the Palestinian Authority to: (a) improve the policy-making environment for tertiary education management, governance, and quality assurance; (b) increase the internal and external efficiency of the tertiary education institutions; and (c) create incentives and provide the basis for improvements in efficiency, quality, and relevance of supply of tertiary education." The PDO in the PAD (page 4) uses similar wording, though it includes as an additional objective improvement in equity in tertiary education institutions.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

Component 1: Strengthening the policy-making role of Ministry of Education and Higher Education, the Council of Higher Education, and the Accreditation and Quality Assurance Committee (appraisal amount: US$ 2.4 million; final amount: US$ 4.2 million). This component aimed to strengthen the capacity of the Ministry of Education and Higher Education, the Council for Higher Education, and the National Accreditation and Quality Assurance Commission to formulate, plan and monitor tertiary education policy.

Component 2: Increasing the internal and external efficiency of tertiary education institutions (appraisal amount: US$ 1.3 million, final amount: US$ 2.5 million). This component aimed to improve the internal and external efficiency of universities and technical colleges by: (i) supporting institutional planning and management, to allow institutions to plan and manage resources in a more efficient way; and (ii) supporting institutional self-assessment and quality improvement, to enhance the capacity of universities and colleges to conduct institutional self-evaluation and develop institutional improvement plans.

Component 3: Improving quality at tertiary education institutions (appraisal amount: US$ 5.5 million; final amount: US$ 14.5 million). This component was to support the creation of a Quality Improvement Fund, which provides competitive grants to tertiary institutions. The goal of the Fund was to provide support to improve the quality of Palestinian tertiary institutions and programs, to make them: (i) relevant to the job market and economic development; (ii) made competitive with international standards; and (iii) capable of developing income-generating programs.

Component 4: Building capacity to improve and expand the student aid program (appraisal amount: US$ 0.8 million; final amount: US$ 0.4 million). The objective of this component was to improve the management and performance of the student aid program, by supporting a Student Revolving Loan Fund to improve its management and governance. The component financed technical assistance to the Fund. The Bank ended its support for this component when UNESCO offered to provide technical assistance to the Fund.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Cost, Financing, and Borrower Contribution. At appraisal, the project was estimated to cost US$ 10 million, financed entirely by a trust fund grant from the World Bank. The European Commission provided EUR 5,000,000 (the equivalent of US$ 5.9 million at the time and US$ 6.6 million at closing) on March 9, 2006. The World Bank provided additional financing through a trust fund grant of US$ 5 million on May 28, 2009. The ICR reports that all financing was fully disbursed by the end of the operation. The Palestinian Authority did not make a financial contribution to the project.

Dates. The project was approved on April 26, 2005 and became effective on June 25, 2005. The restructuring of March 27, 2006 introduced financing from the European Commission to the Project. The restructuring of May 30, 2007 eliminated component 4. The restructuring of May 28, 2009 increased World Bank funding by US$ 5 million and extended the closing date to December 31, 2012. Five additional amendments (pages 6-7 in the ICR) made administrative changes in the project's operation. The project closed on December 31, 2012, two years after the original closing date of December 31, 2010.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

The relevance of objective is rated high.

At appraisal, public support for the system of tertiary education was weak, with rapidly increasing enrollments and heavy reliance on student fees. Declining public finance for universities, coupled with increasing demand, had resulted in declines in quality of services. The system faced challenges related to financial sustainability, efficiency in management of available resources, relevance of quality and supply of education, and lack of equitable distribution of student aid programs. The project's objectives were highly relevant to local conditions at the time of appraisal.

By design, the Bank's strategy for the West Bank and Gaza is flexible, responding to a changing environment and the availability of resources from co-financiers (page 17 of the Interim Strategy Note). Several activities proposed under the Interim Strategy Note focus on areas supported by the project. In addition to continuing to support higher education projects, the strategy includes a focus on job creation, and will include analytic and advisory activities on school-to-work and employment-oriented education programs (pages 17-18 and 23). The project remains highly relevant to the Bank's and Government's work moving forward.

b. Relevance of Design:

The relevance of design is rated substantial.

The project's design included many of the elements that could realistically contribute to achieving the development objectives. The project provided capacity-building support (through Component 1) that would plausibly contribute to improvements in the policy-making environment.

The project's support of external efficiency was supported by Component 3 and its focus on improving the quality of institutions and their relevance to the broader economy. Less directly, Component 2 also supported the goal of external efficiency through its support to self-assessment. The project also aimed at improving the internal efficiency of tertiary education institutions (i.e. the capacity of these institutions to manage themselves efficiently), which was also supported by Component 2 and its activities focusing on institutional planning and managing.


4. Achievement of Objectives (Efficacy) :


The development objectives focus on improving the policy-making environment for tertiary education and increasing the internal and external efficiency of tertiary education as well as improving quality, efficiency, and relevance in tertiary education. In practice, internal efficiency in the second phrase of the objective refers to efficiency in the third phrase. External efficiency and relevance both refer to the same concept. Given the project's objectives and its design, it is difficult to separate the concept of quality and relevance, as quality here refers to academic improvements. This review will therefore consider the project's contribution to (i) improving the policy making environment; (ii) improving internal efficiency; and (iii) improving external efficiency, relevance, and quality.

Improving the policy-making environment

Achievement of this objective is rated substantial.

Outputs

The project supported the development of a higher education strategy, which will serve as the basis of policy-making in the sector. Given the recent creation of the Palestinian Authority and the complicated structure of the sector (with many independent universities and colleges as well as several policy making agencies), this is an important achievement.

The project also contributed to the high number of universities and colleges that have completed institutional self-assessments. The project originally targeted 50 percent of universities and 30 percent of colleges to achieve these self-assessments. At project closing, all universities and around 70 percent of colleges had completed the self-assessment. The self-assessment is important because it is an important component of an accreditation process.

The project supported the creation of the Accreditation and Quality Assurance Commission as well as the Board of the Quality Improvement Fund. These bodies contribute to the overall regulation of the sector and have the potential to improve the system's focus on quality.

The project also provided support to the various policy makers in the sector, particularly the newly created Ministry of Higher Education (pages 14 and 18 of the ICR). The project also started the first sector-wide data collection effort (pages 21-22 of the ICR).

Outcomes

The ICR reports that there has been significant institutional strengthening in the sector, including improvement in the Ministry’s stewardship and supervisory capacity. The ICR does not provide evidence or specific examples of this process (page 18 of the ICR). The Accreditation and Quality Assurance Commission and the Board of the Quality Improvement Funds have provided effective technical supervision (page 9 of the ICR).

The project also created a framework for accreditation with international standards, with a functioning Accreditation Commission (page 18 of the ICR) as well as introduction of the concept of self-assessment. While the ICR reports that the higher education strategy was not approved, the project team reported that the Ministry and the sector use the strategy in their planning and operation.


Increasing the internal efficiency of higher education

Achievement of this objective is rated modest. From the development objective ("internal efficiency...of tertiary education institutions") and the design of the project, it is clear that internal efficiency here refers to the capacity of tertiary education institutions to manage themselves efficiently.

Outputs

The ICR reports that the Bank’s program focused on developing efficient institutions. Component 1 supported efforts to establish a management information system at the institutional level. Likewise, the component supported the establishment of a Higher Education Management Information System that would allow for more data-driven decision making at both the system and institutional levels. The project supported the establishment of Quality Units at most institutions (this also contributes to quality).

Outcomes

The results framework does not have any measures of the internal efficiency. The project team reports that tertiary education institutions are increasingly allocating resources through an evidence-based planning process.

Increasing the external efficiency, quality, and relevance of higher education

Achievement of this objective is rated substantial.

Outputs

The project supported a number of initiatives to increase the external efficiency of higher education institutions (i.e. to make them relevant for the economy) and graduates (i.e. to increase their probability of finding employment). The Quality Investment Fund made competitive grants to 100 percent of eligible universities and 30 percent of colleges (or 50 percent, when counting colleges that received grants in cooperation with institutions), compared to a target of 50 percent of universities and 30 percent of colleges. It is likely that part of this overachievement of the target was due to the increase in resources for Component 3. At the same time, the large number of institutions that finalized the self-assessment also contributed to the increase in grants.

As part of the accreditation system, the project supported the establishment of Quality Units at most institutions. In general, 76 percent of the activities receiving grants were rated sustainable or better by an independent evaluation, suggesting that they were of acceptable quality and had the means to continue operating after the end of the project (page 28 of the ICR). Given the size of the West Bank & Gaza’s economy, these grants were relatively large.

Outcomes

The ICR argues that the project, through its grants and its support for accreditation, played a major role in increasing the connection between institutions and the the labor market, This conclusion is based on an independent evaluation supported by the project. The methodology of this evaluation is qualitative and appears to be rigorous. The percentage of new students that enrolled in priority program areas increased from 32 from 42 percent, compared to a target of 40 percent. This is likely to be partially the result of the increased focus on quality in priority areas (pages 16, 18 and 31-32 of the ICR). .


5. Efficiency:


Efficiency is rated modest.

Neither the PAD nor the ICR attempted to estimate the project’s economic rate of return. This is not surprising given that the project focused on improving the tertiary system through a variety of interventions, including the provision of sub-grants to tertiary education institutions. The economic analysis in the PAD did provide a cost-efficient analysis of Component 4. This was a relatively small component and it is not clear that the project's support would have led to the assumed impact. The component was eventually dropped.

Component 3 provided grants to higher education institutions, the largest single intervention in terms of financing. Due to the decentralized nature of these grants, it is difficult to analyze the efficiency of each individual project. The grants were awarded on competitive basis and there no financial management issues or reports of corruptions.

The project coordinated well with other donors, particularly with UNESCO. The decision to transfer the activities from Component 4 to UNESCO represents a cost savings to the project as UNESCO was willing to carry out the work, allowing the project to reallocate its resources to other activities.

The ICR provides little information on the project's efficiency. The project team reported that the project's administrative cost represented less than 8 percent of project costs. Given that a major function of the project implementation unit was to supervise sub-grants, this represents a reasonable expenditure and is in line with similar projects in the tertiary education sector. There is little other information to show that the project's resources were spent efficiently.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


The project's outcome is rated moderately satisfactory.

The project's objectives remain highly relevant, and the project's design largely responded to these objectives. Achievement of the objective to improve internal efficiency is rated modest. Achievement of the objectives to improve the policy-making environment and improve external efficiency, relevance, and quality is rated substantial. Efficiency is rated modest. Although the project appeared to have reasonable administrative costs, there is little additional evidence that it was efficiently implemented.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:


The project operated in a complicated political context, with uncertainty about the future organization of the Palestinian Authority (page 9 of the ICR). The government is dependent on external assistance and external assistance is dependent on political support from a variety of financiers. In addition, the project's experience has shown that the sector may be affected by political changes within the government (pages 9-10, 11, and 21 of the ICR). The project team reported that the government has shown strong commitment to maintaining the project and there has been considerable political support to the project's objectives. The World Bank has continued its support for the sector as have other development financiers.

The ICR reports that most of the activities supported by grants to universities were sustainable, as they generated sufficient income or had sufficient funding to continue after the project closed (Annex 3).

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The project was developed under difficult circumstances, with a new and weak government, the Palestinian Authority, as well as a post-conflict situation. Despite these challenges, the Bank team performed well in supporting project preparation.

On the technical side, the Bank carried out three studies on higher education from 1997 to 2002, as well as studying the cases of other Bank-financed higher education projects. During preparation, the Bank and the government took advantage of a grant to update this technical work (page 9 of the ICR). As a result, the project's PAD had a serious discussion of alternatives (page 10). The Bank also participated in an extensive series of stakeholder consultations prior to the project's approval (page 9 of the PAD).

However, one shortcoming was the complicated and ambiguous project objectives. While the project objective was fully supported by the government and Bank strategies, its wording was complicated. The Bank could have taken steps to better target the objective.

The PAD included a comprehensive economic analysis, including a well motivated discussion of the potential changes in enrollment and wages as the result of the project. The economic analysis also discussed the potential social benefits from investing in tertiary education. Given the complexity of the issues involved, it is not surprising that the PAD did not attempt a cost-benefit analysis of all elements of the project (Annex 9 of the PAD).

Financial management risk was rated high due to the high risk associated with the competitive grant component. To reduce this risk, the preparation team carried out a sample assessment of three higher education institutions. The team also worked with the Project Coordination Unit to review the financial management capacity of all higher education institutions to determine their procurement and financial management capacity (Annex 7 of the PAD).

The preparation team carried out a detailed social assessment, which determined that there were no major safeguard issues. The social assessment served to strengthen the project's design by identifying household factors related to the demand for higher education (Annex 10 of the PAD).

The preparation team reviewed all of the required “readiness filters” and confirmed that the project was ready for implementation and did not require any policy exceptions (page 19 of the PAD).

Quality-at-Entry Rating: Satisfactory

b. Quality of supervision:

Despite the challenges associated with the implementation of the project, the Bank team was able to supervise the project effectively and provided value added.

The Bank supervision team was able to manage a substantial increase in resources, both from the Bank’s Trust Fund and the European Commission. In addition, the Bank transferred the responsibility of its support to develop student loans to UNESCO, which represented both a savings in resources and improved donor coordination.

One shortcoming is related to the results framework. Although the project received substantially more resources than in the original appraisal (particularly for Component 3), it did not appear to adjust its results framework to accommodate these changes. The different restructurings and amendments could have facilitated these changes.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

In this context, the government refers to the Ministry of Education and Higher Education and its successor, the Ministry of Higher Education, as well as the Palestinian Authority.

There were a number of shortcomings in the government’s performance. Of particular importance was the lack of approval of the National Tertiary Education Strategy.

In addition, there was a lack of continuity and inadequate staffing. For example, the Quality Improvement Fund saw six directors in seven years (page 21 of the ICR). Likewise, there was often poor coordination between the various government offices (page 21 of the ICR). According to ICR, the Project Technical Committee stopped holding meetings. Likewise, the Ministry stopped providing a strong oversight function. This may have slowed down the approval of the Higher Education Strategy and delayed the accreditation process (page 21 of the ICR). The ICR also notes that there were often problems with communication among various units and the Committee (page 9 of the ICR).

Government Performance Rating: Unsatisfactory

b. Implementing Agency Performance:

The implementing agency was the Project Coordination Unit (PCU), which was responsible for managing the project with the cooperation of three distinct government bodies as well as a number of higher education institutions. The PCU managed complicated relations with the government and tertiary education institutions. For example, it supported most institutions to carry out self-assessments, and it supervised two evaluations. According to the ICR, the PCU was effective in carrying out its procurement and financial management functions (page 21) as well as carrying out the supervision of a large number of grants and projects (page 9).

The PCU maintained continuity from the previous Bank tertiary education project and kept the Bank up to date by providing necessary reports in a timely fashion (pages 9 and 21 of the ICR).

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

Under the initial project design, the Project Coordination Unit was responsible for all project-level aspects of M&E. The PCU was to provide periodic reports to the Project Technical Committee as well as to the Bank. Participating higher education institutions were to provide annual reports of required data (page 10 of the ICR). The PAD adequately identified sources of data to be collected as well as the frequency of collection. For quantitative targets, the PAD included baselines where appropriate. Several indicators (such as the start of self-assessment) were new initiatives, and the effective baseline was therefore zero (Annex 3 of the PAD).

The project's results framework had a number of shortcomings. It did not have a good set of outcome indicators that were aligned with the development objective. For example, there was no clear way to measure whether policy making, external efficiency, or internal efficiency improved. Likewise, the intermediate outcome indicators did not provide thorough coverage of the expected impact of the project's activities.

b. M&E Implementation:

The Project Coordination Unit collected the agreed data from institutions and compiled the required reports to the government and the Bank. The PCU also supervised two external evaluations (pages 11 and 18-19 of the ICR). Higher education institutions were strengthened in their capacity to collect data and carry out self-assessments (page 17 of the ICR). While the PCU did a good job in collecting some data, there were many areas that did not receive attention, largely because the design did not include many key variables.

a. M&E Utilization:

The utilization of M&E was mixed, in large part due to the weakening of the Project Technical Committee, which led to a greater distance between the Ministry and the project (page 11 of the ICR). At the same time, the project did continue to make use of information, such as the self-assessment of institutions. The project team reported that data generated by the project's information management system was used to improve the allocation of resources within tertiary education institutions.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

During appraisal, no safeguards were triggered. The ICR reports that the project was in compliance with the Bank's safeguard policies.

b. Fiduciary Compliance:

According to the ICR, the PCU was effective in carrying out its procurement and financial management functions (page 21) as well as carrying out the supervision of a large number of grants and projects (page 9). The ICR (page 22) indicates that audit reports were submitted in a timely manner, but it does not state whether audits were unqualified.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
The project's objectives remain highly relevant, and the project's design largely responded to these objectives. Achievement of the objective to improve internal efficiency is rated modest. Achievement of the objectives to improve the policy-making environment and improve external efficiency, relevance, and quality is rated substantial. Efficiency is rated modest.  
Risk to Development Outcome:
Moderate
Significant
While the Bank and other partners have remained committed to the sector, there are significant political risks. 
Bank Performance:
Satisfactory
Moderately Satisfactory
The project had strong quality at entry, building on previous Bank work in the sector. The Bank provided adequate supervision, although there were several missed opportunities to update the results framework.  
Borrower Performance:
Satisfactory
Moderately Satisfactory
The implementing agencies' performance was satisfactory, while the government's performance had significant shortcomings.  
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The ICR reports several lessons (pages 22-23). These include:
    • A well structured and designed quality improvement fund can promote relevance and innovation. In the project, these funds were crucial in improving the relevance of higher education.
    • Having the right mix of skills is essential. Each organization that participates in a project (Bank, government, implementer) needs to have a combination of operational and technical skills in order to be effective.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR does a reasonable job in explaining the project's design, implementation history, and outcomes. Given the nature of the available data, the ICR often relies on qualitative evidence to make its case. The ICR does not explain whether the project was efficient (provided good value for money) and does not present sufficient evidence about the project's impact on students. The ICR would have also benefited from a better developed discussion of the project's outputs.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions