|1. Project Data:
ICR Review Date Posted:
|Higher Education For Relevance And Efficiency
Project Costs(US $M)
Loan/Credit (US $M)
Cofinancing (US $M)
Board Approval Date
|Tertiary education (99%), Central government administration (1%)|
|Education for the knowledge economy (40% - P)
Law reform (20% - S)
Decentralization (20% - S)
Administrative and civil service reform (20% - S)|
||ICR Review Coordinator:
|Erik A. Bloom
||Judyth L. Twigg
||Christopher D. Gerrard
|2. Project Objectives and Components:|
The project’s Development Credit Agreement (p. 17) states that the project's objective was:
The objective in the Project Appraisal Document (p. 5) has the same wording.
“to create an enabling environment for the evolution of autonomous and accountable public higher education institutions, and to develop effective support mechanisms for the improvement of the quality, relevance, efficiency and equity of higher education.”
b. Were the project objectives/key associated outcome targets revised during implementation?
The project had three components:
Component 1: Higher education system reform and oversight (originally estimated at US$8.8 million with actual disbursement of US$2.1 million, which was 24 percent of the planned amount) was to support the government in implementing its Higher Education Long Term Strategy 2003-2010 by promulgating a new higher education law, strengthening the Ministry of Education and Culture's (MoEC's) Directorate General of Higher Education management and administration, assisting with developing an institutional accreditation program, and developing a strategy to revitalize the Open University.
After a Level II restructuring (2009), several activities were added, including capacity building support for a Global Distance Learning Network to train and mentor technical staff of the Indonesia Higher Education Network (INHERENT), especially on video conferencing and on-line training; and scholarships to Open University staff to strengthen their distance learning and online training techniques.
Component 2: Grants to improve academic quality and institutional performance (originally estimated at US$97.9 million with actual disbursement of US$82.7 million, which was 85 percent of the planned amount) was to support improving the quality of higher education through targeted investments and by improving management capacity at Higher Education Institutions (HEls). Some of the component financing was cancelled due to government restrictions on implementing one of the grant programs as well as a lower than expected grant awards in another grant program (page 9 of the ICR).
Component 3: Project Management (originally estimated at US$7.8 million with actual disbursement of US$7.7 million, which was 98 percent of the planned amount.) was to provide financial support to the Implementation Unit to implement the project. After the Level II restructuring (2009), this support also efforts to expand the role of the Implementation Unit to build procurement, financial management, and human resource management capacity at HEls to run workshops, short training courses, and on-site mentoring, monitoring and supervision.
d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
At the time of appraisal, the project was estimated to cost US$114.5 million. The actual cost of the project was US92.5 million, which represents 81 percent of the planned amount. All of the major activities added to Component 1 at the 2009 restructuring appear to have been carried out as planned, although often with alternative sources of financing (ICR, p. 26). This led to significant cost savings.
The Bank provided a US$30 million IDA credit, of which 99.9 percent was disbursed. It also provided a US$50 million IBRD loan, of which 80.3 percent was disbursed.
A total of US$6.9 million was formally cancelled from the IBRD loan, largely due to changes in Component 1. An additional US$1.9 million was left unspent from IBRD and cancelled at the end of the project. A small amount (US$ 45,000) was left unspent from the IDA credit.
The government agreed to finance US$34.5 million equivalent to support the project. At the time of project closing, it had provided US$22.4 million equivalent, equal to 64.9% of the initial budgeted amount.
By the end of the project period, the government provided 84.5% of planned financing for Component 2. The shortfall was due to the Ministry of Finance's decision to stop block grants. The government also provided only 24.1% of planned financing for Component 1. This was largely driven by the decision to reduce the size of the component, from US$8.8 million to US$6.1 million. Actual expenditures on this component were only US$2.1 million, and the government actually contributed a larger share than originally planned (ICR, pp. 51-52).
The project was approved by the World Bank Board of Directors on June 9, 2005 and became effective on December 20, 2005. The project was restructured (Level II restructuring) once, on September 24, 2009. The restructuring was largely based on observations made during the mid-term review, held in the same year, that led to changes in the project's implementation structure (ICR, pp. 8-9), and also in response to the government's 2009 decision to stop block grants to public HEIs and shift instead to line-item budgeting, as required by the Indonesia budget law.
The original closing date was June 30, 2011, but this was extended as part of the restructuring to December 31, 2012. The project closed on the latter date.
|3. Relevance of Objectives & Design:|
a. Relevance of Objectives:
The relevance of objectives is rated high.
At the time of approval, the project was based closely on the government's tertiary education strategy (Higher Education Long Term Strategy) as well as the Bank's Country Assistance Strategy (CAS), which focused both on improving human capital and strengthening public institutions. The project was designed to support the government's efforts to strengthen the country's tertiary education system (ICR, p. 5), alleviating shortages of higher-level skills that were slowing investment and the absorption of new technologies.
The project remained highly relevant at the time of project closing. The government continues to include the goals of the reform in its development strategies and passed a law in 2012 that codified many of the project's objectives, replacing a previous law that was ruled unconstitutional by the Constitutional Court (ICR, p. 27). While the government has decided to stop borrowing for education projects, the project remains relevant to current World Bank strategy in a number of areas, including: (i) institution building for the public sector; (ii) improving education and skills for the labor force; (iii) enhancing equality of education opportunities; and (iv) strengthening human resources in research and development. In many cases, the Bank plans to support these activities with technical assistance when lending is not an option (Country Partnership Strategy 2013-2014, pp. 20-21).
b. Relevance of Design:
The Relevance of Design is rated high.
The project had two different objectives: (i) improving the policy environment, and (ii) providing support to universities' mechanisms for improvements in the quality, relevance, efficiency and equity of higher education ("university" is used to indicate any higher education institution). While these goals are related and may be complementary in many cases, they require quite different sorts of interventions.
In order to improve the policy environment, the project focused on strengthening the legal framework for higher education as well as strengthening the Ministry's stewardship capacity (Component 1). Support for the legal framework included technical assistance to the Ministry. The project also supported the Ministry's financial management capacity and the autonomy of the accreditation system, as well as studies and data collection to monitor the higher education system and its impact on the life of graduates.
The project also provided support to universities. Central to this was the provision of grants to universities, through Component 2. The design of this component was based on policy recommendations for higher education and included competitive grants available to both public and private universities. These grants were expected to cover a wide range of areas, including the quality, relevance, and equity of higher education. The grants were designed to include support for laboratory equipment, staff development, and studies (quality and relevance) as well as support for community development and scholarships (equity).
In addition, the project also supported three grant programs to support university governance in different types of institutions, including autonomous public universities and non-autonomous public universities, using a mix of competitive grants and performance-based grants. This design showed appropriate flexibility in the face of different types of university structures. Most of the support focused on improving university administration, through investment in financial and procurement management as well as human resource management, which are intended to improve efficiency at the university level. The project also provided direct support to universities to strengthen their financial management capacity to manage the grant program, through Component 1.
Overall, all of the project's planned activities were closely aligned with the project's objectives or administration and represent careful attention to both aspects of the objectives.
|4. Achievement of Objectives (Efficacy) :|
Create an enabling environment for the evolution of autonomous and accountable public higher education institutions.
Achievement of this objective is rated substantial.
The project supported a number of policy initiatives that contributed to accountability in higher education in Indonesia. These are reflected both in outputs supported by the project as well as outcomes related to increased autonomy, accountability, and good governance of the institutions in the sector.
The project supported the approval of Higher Education Laws in 2009 and 2012 that established a governance framework for universities. The project provided this support by financing seminars, study tours, and reports. The first law was overturned by the Constitutional Court and replaced in 2012 with a new law that had a similar objective and responded to concerns raised by the Court (the ICR does not describe the Court's specific concerns). The government reports that the project played an important role in supporting the development of both laws (ICR, p. 6).
The project provided training for university and ministry staff in financial management. It also supported training for procurement staff at the university level, which led to the certification of procurement staff in most of participating universities (28 of 29 participating universities had 50% or more of procurement staff certified in accordance with national certification policy, meeting the target).
The project supported a number of studies and surveys to provide accountability. This included a tracer study of graduates that was developed and implemented but has not been yet fully institutionalized. Tracer studies are often seen as important measures of accountability.
The project supported efforts to create an accreditation framework to monitor the quality of universities. In addition to the framework, the project provided technical assistance to universities to participate in the accreditation process.
Through Component 2, the project provided grants for good governance to a number of public universities (including autonomous universities). Although this is usually associated with the second objective (see below), these activities are also appropriately associated with the first objective. They included five results-based grants to autonomous universities (out of six proposals), proposal-based grants to all autonomous universities, as well as 29 grants (out of 115 proposals) for non-autonomous public universities. The ICR Annex Table 2.5 (pp. 43-44) outlines the activities supported by the grants. According to the ICR (p. 44), these activities were largely carried out as expected.
There was an increase in both the capacity and transparency of universities. This contributed to their autonomy and accountability. There are several indicators of this process:
- Unqualified audit reports in seven universities, against a target of five by the end of the project.
- 86 percent of procurement by participating universities occurred during the bid validity period, compared to a target of 90 percent. Despite the fact that the project missed its target, this is an impressive achievement given that at the start of the project, most universities had no procurement capacity.
- Likewise, 83 percent of participating universities published the results of their major procurement. While this does not meet the target of full disclosure by all universities, it represents an important advance from a situation where no procurement was publicized.
- 52 public and 42 private universities were accredited. This represents 55 percent of public universities and substantially exceeds the target of 5 percent of public universities accredited. Approximately 1.3 percent of private universities were accredited; it is not clear if the project had a target for private universities (ICR, p. 30). Given that accreditation is new to higher education in Indonesia, this represents a substantial achievement.
- Performance-based contracts were awarded to five out of six autonomous universities. The creation of this type of contract is an important outcome to promote autonomy. The support to 29 other public universities may have also strengthened their capacity to act in a more autonomous manner.
Develop effective support mechanisms for the improvement of the quality, relevance, efficiency and equity of higher education.
Achievement of this objective is rated modest.
This objective was largely supported by the grant programs under Component 2. These programs were affected by a decision of the Ministry of Finance to prohibit the use of block grants in 2009 and a decision of the Constitutional Court to overturn the Higher Education law in 2010. By the end of the project, 84.5 percent of total amount of Component 2 was disbursed, and the project was able to continue with other forms of grants.
Some of the outcomes related to the first objective can be considered outputs for this objective. They include: (i) unqualified audit opinions for seven higher education institutions; (ii) university procurement carried out in a timely fashion; (iii) training of procurement staff; and (iv) publishing of contracts financed under the grant. In addition:
- 38 competitive grants (averaging US$1.2 million) were awarded to mostly public universities. These grants covered a number of areas from seven cost categories: (a) staff development; (b) technical assistance; (c) research and studies; (d) community development; (e) procurement of goods for a variety of capital goods and educational inputs; (f) promotional activity aimed at secondary school students: and (g) scholarships for poorer applicants. ICR Annex Table 2.2 (p. 38) reports the allocation and actual expenditures under each of these cost categories.
- The project awarded five performance-based grants to autonomous universities. While these grants contributed to the strengthening of autonomy (see above), they also contributed to the quality and equity of education in these universities.
- The grade point average increased from 2.99 to 3.17 (compared to a target of 3.13) between 2009 and 2012 in universities participating in the grant program. While this is an important improvement, it is difficult to show causality with the project due to the lack of comparison data. It is also difficult to show that this reflects a real change as opposed to "grade inflation." The project's outputs (through grants) included activities that may plausibly have contributed to improving quality and student learning, including improved equipment, better-trained staff, and research.
- The length of time necessary for graduates to find employment declined from 8.5 months to 5.8 months (compared to a target of 7.6 months). This is a measure of both relevance and efficiency. The use of a tracer study increases the credibility of this measure; however, the lack of data on the performance of graduates from non-participating universities as well as general changes in the labor market make it difficult to establish causality.
- The number of students who received need based scholarships increased from 8 percent to 12 percent (compared to a target of 12 percent). Studies have shown that financial support plays a major role in increasing access to universities, and this was the likely outcome in Indonesia.
- The average time to graduate dropped from 56.7 months to 51.6 months (compared to a target of 54 months). This represents an increase in internal efficiency. This increase is likely due to changes within the universities, which in turn may have been the result of the investments supported by the grants.
Overall, there was a significant improvement in all of the indicators associated with this objective. While the project's design included a causal chain that could plausibly have led to these outcomes, the available data make it difficult to compare the results with non-participating universities.
The project's efficiency is rated substantial.
At the time of appraisal, the project team carried out a detailed economic analysis based on careful study of economic conditions, the labor market, and the project's design. The analysis clearly laid out its assumptions and carried out a sensitivity analysis. The economic analysis showed an estimated internal rate of return from 32 to 50 percent.
The ICR had an equally comprehensive economic analysis, focusing on the project's actual performance and its impacts given current economic conditions. These results show a similar rate of return (30 percent). These estimates are reasonable given the relatively high rates of return associated with higher education in Indonesia (PAD, p. 77, and ICR, p. 56) and the broad impact of the program on the higher education sector.
The program closed 18 months later than planned, which does not represent a serious shortcoming. Due to the change in the legal framework, it was not possible to carry out large parts of Component 2. The government and the Bank agreed to cancel parts of the loan rather than trying to find another area in which to invest the remaining resources. This was a prudent use of Bank resources.
a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:
* Refers to percent of total project cost for which ERR/FRR was calculated