|1. Project Data:
ICR Review Date Posted:
|Mx (apl I) Tertiary Educ Student Ass
Project Costs(US $M)
Loan/Credit (US $M)
Cofinancing (US $M)
Board Approval Date
|Tertiary education (80%), Sub-national government administration (10%), Central government administration (5%), Secondary education (5%)|
|Education for the knowledge economy (33% - P)
Indigenous peoples (17% - S)
Export development and competitiveness (17% - S)
Administrative and civil service reform (17% - S)
analysis and monitoring (16% - S)|
||ICR Review Coordinator:
|David W. Berk
||Judyth L. Twigg
|2. Project Objectives and Components:|
a. Objectives:The program (over 10 years in three overlapping phases) and the project (financed by the first of three planned APLs) had identical objectives as follows.
According to the Loan Agreement (p. 20), the objective was to “support the Government strategy to foster the sustainable and equitable and efficient expansion of tertiary education through student assistance through the development of a coherent student assistance system consisting of student loan and grant programs, compensatory interventions for disadvantaged students, and supporting the national tertiary education policy."
The PAD contains two differing versions of the PDO. On page 6, “the proposed APL would support the Government’s tertiary education strategy by strengthening the Government’s student assistance program to foster the sustainable, equitable and efficient expansion of tertiary education.” However, on pages vi and 8, "the proposed program supports the Government strategy to foster the sustainable and equitable expansion of tertiary education through student assistance.”
In assessing the project objectives, this ICR Review follows the Loan Agreement by using a four-part PDO: “fostering the sustainable, equitable and efficient expansion of tertiary education.” This broader set (including efficiency) seems to have been the original intention: there are related project components or actions; the restructuring (see next paragraph) paper repeats the broader set; and the ICR includes assessment of efficiency gains through the project.
Those portions of the Loan Agreement and PAD statements of the PDO that followed the words "through" and "by" are not objectives, but means to reach the objectives, namely outputs and inputs. As a consequence, the changes during project implementation - dropping the student loan component - were not changes in the PDO.
Key associated outcome targets for the end of the three-APL series were:
1. Increased numbers of graduates from tertiary education (expansion )
2. Increased share of enrollments for students from households in the two lowest income quintiles (equity objective)
However, there were no key outcome targets measuring the sustainability and efficiency objectives.
b. Were the project objectives/key associated outcome targets revised during implementation?
If yes, did the Board approve the revised objectives/key associated outcome targets?
Date of Board Approval: 03/01/2010
c. Components: 1. Support for the National Scholarship Program for Higher Education (PRONABES), a federally-designed program in which states participate voluntarily and share the financing 50-50 with the federal government; and for development of a national regulatory framework for tertiary education student assistance (appraisal estimate US$281.23 million, actual US$284.09 million): (i) Scholarships for an estimated 400,000 tertiary students from poor families, with the Bank financing 58% of the federal share or 29% of the total scholarship budget. (ii) Building federal regulatory and supervisory capacity to build and oversee an efficient and financially sustainable tertiary education system.
2. Analysis and institutional strengthening (appraisal estimate US$1.73 million, actual US$0.76 million): (i) Creating a database on the labor market performance of tertiary graduates and institutions. (ii) Studies providing the analytical foundation for strengthening tertiary education policy: income-contingent reimbursable grants, financing alternatives, equity, quality assurance mechanisms, links to the productive sector. (iii) Evaluation of student assistance programs.
3. Support for disadvantaged students (appraisal estimate US$5.92 million, actual US$4.03 million): (i) Early interventions to assist talented low-income students in upper secondary education (pilot). (ii) Expansion of the Program for Assistance to Indigenous Students in Tertiary Education Institutions (PAEIIES), which was managed by the National Association of Universities and Higher Education Institutions (ANUIES), in accordance with the project’s Indigenous Peoples Development Plan.
4. Support for state-level student loan programs (appraisal estimate US$10.58 million, actual $0): Program in one state (expected to be Quintana Roo) for loans and institutional strengthening. This was to be financed by a separate Bank loan for the project to the National Bank of Public Services and Works (BANOBRAS).
5. Promotion of private investments in student loans (appraisal estimate US$0.12 million, actual US$0): Design of ways to attract private capital and corresponding institutional arrangements, to be based on study of alternative financing for higher education, with implementation in later phases of the program.
d. Comments on Project Cost, Financing, Borrower Contribution, and DatesProject Costs: Total project cost for phase I was originally US$300 million but diminished by US$10 million. The main reason was that component 4 for student loans and component 5 for private participation were removed from the project. Also, the Government implemented component 2(i) with other funding, achieving the project objective. Funding from an alternative source compensated for a late start to programs for indigenous students. Savings in components 2 (i), 3 (i) and 3 (ii) were reallocated to provide additional scholarships.
Financing: The original Bank contribution to phase I was to be US$180 million through two loans. The US$171 million Bank loan to the Government was fully disbursed. The second Bank loan, for US$9 million to BANOBRAS for student loans, was withdrawn 18 months after Board approval. BANOBRAS hardened its proposed terms for on-lending the Bank loan, and BANOBRAS and the newly elected authorities in the state of Quintana Roo never reached final agreement on legal and financial terms. No other state with implementation capacity expressed sufficient interest in such a program within the allotted time frame. The advent of the financial crisis may have contributed significantly to reduced interest in loan programs. The second APL was planned to be all-loan, and the third one a mixture of loans and some grants. Given its loss of interest in student loans, the Government decided not to proceed with the second and third proposed APLs.
Borrower Contribution: The Borrower’s original contribution to project costs was to be US$120 million. Its actual disbursement was US$119.3 million.
Dates: A slow start to the project under a newly elected government, and delays in initiating component 3(ii) because of coordination problems between the Government and the umbrella organization involved, required postponing the original closing date of March 1, 2010 by 22 months to December 31, 2011, leading to a project duration of six years.
|3. Relevance of Objectives & Design:|
a. Relevance of Objectives:High.
At approval the Government’s national development plan and education sector strategy both covered the period 2001-06. With the advent of a new administration in December 2006 and the financial crisis in 2008, its immediate priorities were restated under a Ten Point Program.
Keeping the expansion of tertiary education sustainable and efficient - and therefore not “budget-busting” - remains highly important. Making it more equitable was central to the Government’s national development plan’s “equality of opportunities” pillar, and to its education sector strategy that emphasized increase in opportunities to reduce inequality.
The Bank's Country Partnership Strategy for FY08-13 focuses on accelerating growth; improving competitiveness; promoting social inclusion and reducing poverty; developing infrastructure and assuring energy security and environmental sustainability; and strengthening institutions. The program’s and project’s focus on sustainable, equitable, and efficient expansion of tertiary education contributed particularly to the competitiveness objective and that of reducing poverty.
The sustainable, equitable, and efficient expansion of tertiary education was highly relevant to Mexico’s long-term development prospects, in particular its international competitiveness, and also to social inclusion and poverty reduction.
b. Relevance of Design:Substantial.
The project’s causal chain to achieve the objectives was:
Expansion would result from an increased number of scholarships, student loans, and increased “terminal efficiency” (reduced rates of dropout and repetition) in tertiary education. The three aspects of expansion targeted by the program and project would have the following results chains. Sustainability would result from student loans. Equity would result from the targeting of scholarships and from measures to support indigenous students. Efficiency would result from labor market information on successful institutions to attract applicants and from increased “terminal efficiency” (on-time graduation) in tertiary education.
Given that the ERR of the project (Section 5 below) depended on this increased "terminal efficiency," i.e. on reduced repetition and dropout rates of students, there should have been additional (non-financial) support to all PRONABES students under the project, not just to indigenous students.
The omission of student loans during project implementation adversely affected the likely sustainability. It thus weakened the rating somewhat, but not enough to downgrade.
|4. Achievement of Objectives (Efficacy) :|
This review rates separately the expansion objective, as well as three aspects of that expansion: sustainability, equity, and efficiency.
The attribution of results in this section is solely to the project, since the project funded about one-quarter (and directly induced the funding of another quarter) of PRONABES’s expansion.
Foster the expansion of tertiary education: Substantial
Total tertiary enrollments were 2.2 million in 2003. They increased by about 700,000, or about one-third, over the project life (information from project team). The contribution of the project cannot be derived directly from the ICR.
PRONABES-financed students were 6% of the total in 2003. The number of students with scholarships provided by PRONABES increased from 161,787 in 2005 to 315,352 in 2011 (information from project team). The PRONABES increase was equivalent to about 20% of the increase in total enrollments. The total scholarship-years financed over the project life were about 1.6 million. About one-quarter were financed by the project as legally defined (Bank-plus-federal funding), and another quarter by states that were required to match the level of Bank-plus-federal funding under the project. The other half of PRONABES funding was provided by unanticipated major increases in both federal and state budgets for PRONABES scholarships. PRONABES’s total budget (federal plus states) went from MX$1.67 billion in 2005 to MX$3.14 billion in 2011.
The remaining 80% of the overall tertiary enrollment increase appears to have been financed almost entirely by an unanticipated doubling of federal and state budgets for tertiary education between 2006 and 2011 (from MX$43.2 billion to MX$80.9 billion).
The number of tertiary graduates increased from 359,635 in 2004-2005 to 475,584 in 2010-2011, effectively meeting the PAD’s intermediate indicator target of 478,000 for that year.
Project (APL I) funding made a significant contribution to this increase as shown above. The planned contributions from the expected APL II and III by that date toward achieving the target did not materialize. But the institutional strengthening of PRONABES under the project (see next section) gave federal and state governments the confidence to channel double the amount of project (APL I) funding through PRONABES in this period. It is hard to imagine that the actual national graduation results would have been achieved without the expansion of PRONABES, which in turn was to a considerable extent attributable to the existence of the project.
Foster sustainability in tertiary education: Substantial
Tertiary education enjoyed strong political and social support during the project period. This commitment translated into a doubling of federal and state budgets for tertiary education and within them for PRONABES, as detailed above. PRONABES was one of the few programs that saw a funding increase during the financial crisis. These increases have been sustained up to the present. Through the years, more academic institutions have joined the program.
On the other hand, interest in Mexico in student loans was extremely limited during this period. The planned US$9 million in student loans under the project did not materialize. The economic downturn in 2008-2009 further reduced interest.
The project supported considerable institutional strengthening, in particular increases in the political and financial sustainability of PRONABES, through support to its Annual Meetings and -- most importantly -- to development of its financial system and management information system. These furthered communication and compliance between the federal and state governments, and enabled external dissemination of evaluations of PRONABES. The project also supported the sustainability of the Academic Support Units for indigenous students at participating institutions.
The increases in tertiary education budgets appear to reflect a strong social demand and consensus in favor of government spending on tertiary education in general. Such spending can be expected to be sustained in the future.
The improvements under the project and the results have made PRONABES a more credible implementing agency for government policy. PRONABES in its strengthened form represents a major resource for tertiary education promotion in the future.
For purposes of this ICR review, the social and governmental support for tertiary education spending outweighs the abandonment of the student loan component under the project and APL II and III.
In the government’s plan for continuing operation of the project, it has committed to continue promoting increased financing to sustain and expand PRONABES. The current education strategy mentions the government’s intention to increase the coverage of PRONABES in all tertiary education subsystems. Seventeen of 24 universities that participated in support for indigenous students (PAIIES) have indicated that they will maintain the program with their own resources; 12 of them have agreed to provide at least 70% of the financing received under the project. This may be optimistic. However, any shortfall would only have a limited impact on overall sustainability.
The fact remains that the long-term financial sustainability of the tertiary education system will require some contributions from beneficiaries, as originally proposed under the project and program. It is encouraging therefore that the government of Mexico continues to realize this, and has recently begun tentative exploration of some options in this area. But interest in student loans declined considerably during the financial crisis. There is no guarantee that a new contribution scheme will materialize any time soon.
Foster equity in tertiary education: Substantial
The promotion of greater equity was an important plank of national and education sector plans and strategies.
The primary target group under the project was economically and socially disadvantaged tertiary education students.
The annual numbers of PRONABES scholarships increased considerably as detailed above. Targeting, with supporting rules and the use of management information systems, was sharpened progressively. 100% of beneficiaries came from households with income below four times the minimum wage by project closing, versus 96% at the start, and this also meant that 100% of them were from the two lowest income quintiles.
Starting in 2007, priority has been given very successfully to former recipients of the Oportunidades conditional cash transfer program for lower-income families and to indigenous students. For the indigenous students program, there was a substantial delay in establishing the contract between GOM and the university association ANUIES that manages the program. As a result, Bank funds arrived only after a delay of three years. However, unanticipated continuing Ford Foundation (US) assistance permitted reaching 9,600 students by then. The project then supported existing students and added almost 2,400 more, so 11,918 were assisted instead of the originally envisaged 3,500.
Key outcome target #2 was for an increase in the share of national tertiary enrollments accounted for by students from households in the two lowest income quintiles. There was an increase from the baseline share of 10% to the 2010-2011 share of 20.6%, meeting the informally revised target of 20%.
For ex-Oportunidades and indigenous students there were intermediate results indicators, with the following outcomes (% of PRONABES beneficiaries):
2004-2005 Target 2010-2011
Ex-Oportunidades 8.7 10 17.9
Indigenous 6 10 12.6
While there was a Government policy in favor of increased equity in tertiary education, which would likely have affected PRONABES’s allocations in this period even in the absence of the project, the project resulted in better information and most likely in farther and faster shifts in allocations than would have occurred without it.
The policy of equity promotion is likely to continue in future, Any shortfall in funding the indigenous student support program could produce shifts within the poor student population rather than reductions.
Foster Efficiency in Tertiary Education: Modest
The PAD quoted analysis showing that the private return to tertiary education in Mexico as of the early 2000’s ranged from 16%, taking account only of earnings foregone while studying, to 10% including also private direct costs. The social return including also public direct costs was 7%, which is acceptable. These estimates did not take into account other benefits (positive externalities) from investment in tertiary education, such as (according to the PAD) improved technology absorption. These rates of return would be improved if repetition and dropout rates diminished as intended under the project.
Increasing the proportion of students who stay in tertiary education and graduate was an important objective and benefit of the project. However, there was inadequate provision in the project activities to encourage and enable students to achieve this. For the majority of students who were not indigenous, the only measure was that achieving a satisfactory grade point average was a condition of renewal of their scholarship each year. Indigenous students did receive non-financial assistance through the PAEIIES program.
Data collection methods and the six-year project life have not yet permitted PRONABES to collect data on the “terminal efficiency” (proportion of a student cohort that graduates in five years). Instead, the year-to-year retention rate has been used.
There was a modest increase during the project life in the efficiency with which the tertiary education system as a whole turned numbers enrolled into numbers of graduates. The tertiary graduates-to-enrollments ratio in a given year increased from 13.5% in 2003 to 16.5% in 2011. The project contributed to this outcome.
Under the project, the proxy for “terminal efficiency" -- the year-to-year retention rate -- was estimated at 65.3% in 2003-2004, and the target for 2010-2011 was 75%. The actual figure was 87.3% in 2010-2011, a considerable increase. Nevertheless, this figure implies that approximately 57% of the cohort of PRONABES students who started in 2010-2011 will complete their studies in 2014-2015, still an undesirably low proportion. The increase in the retention rate was not enough to lift the economic rate of return to a satisfactory level (see Section 5).
Indigenous students participating in PAEIIES also improved their year-to-year retention rates, though by less than was targeted.
PRONABES increased its efficiency during the project period and kept its administrative costs at less than 1% of the total scholarship budget, through a new financial system and strengthened management information system, online applications, reduced paperwork for informal sector and rural students, and arrangements to disseminate information and coordinate activities with the Oportunidades program.
Only the improvements by indigenous students would probably have occurred without the project.
Progress Toward Objectives of Entire APL Program
The program (APL I-III) and its first phase (APL I – the project) had the same objectives, so that progress under the project represents progress towards the program objectives. Generally speaking, progress was satisfactory, except as regards student loans, which were dropped from the project, and which contributed to the Government’s decision not to pursue APL II and III. This affected the long-term sustainability of the tertiary education system. However, more encouragingly, the government has recently begun again to explore options in the area of beneficiary contributions.
a. Economic Rate of Return:
The PAD's economic analysis assumed that all project benefits came from increasing the “terminal efficiency” of PRONABES beneficiaries, defined as the proportion of a cohort of students that graduates in five years. This increase produces more tertiary graduates (than without the project) who benefit from increased lifetime earnings compared to secondary graduates. The PAD's sensitivity analysis showed how much this proportion of "on-time" graduates must increase to produce a certain economic rate of return, but the PAD did not estimate a single ex ante ERR for the project based on this analysis.
The ICR (pp. 32-3) reports a “terminal efficiency” achieved of 63% (based on the latest available figures, which are for year-to-year retentions), hence a project ERR on the PAD basis of between 3.6 and 4.5%. It states that this justified the project from an efficient use of capital perspective. This review disagrees, considering this an inadequate economic rate of return.
The ICR states that other benefits should also be taken into account, in particular the values of increasing equity and of improving project administration. It makes the following analysis. A proportion of PRONABES graduates and their eventual families can be expected to escape poverty, with substantial savings to future government budgets for anti-poverty programs (in this calculation, programs of conditional cash transfers and free health insurance); the ICR assumes that 50% of graduates/families so escape (information from TTL). The ICR does not provide the detailed calculation, but states that adding these future government savings to the benefits in the PAD analysis yields a final project ERR in the ICR of 8.3 to 16.4%, which is acceptable to very good.
However, this approach is not acceptable. Education projects evaluated by IEG only include benefits that are directly linked to the objective and activity. Also, the expected government savings on other anti-poverty programs may not materialize. While all PRONABES students came from households in the bottom two income quintiles, not all were eligible for other government anti-poverty programs; for example, only 18% were from households benefiting from the Oportunidades program. The programs do not necessarily reach all eligible households now, so removing some households might not reduce total expenditures. Also, secondary graduates are also not generally poor, and some who are poor will continue to escape in the without-project case. Finally, projects in other sectors also raise beneficiary incomes, but the type of positive externality cited in the ICR for this project (savings on future government anti-poverty programs) is not included in their ERR calculations.
There were two other project achievements in the area of efficiency. PRONABES kept its costs below 1% of its scholarship portfolio, an excellent level. It also simplified the application process, saving students and their families considerable time and effort.
This ICR review considers that the inadequate economic rate of return outweighs project achievements in the area of value for money.
Efficiency Rating: Modest
a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:
* Refers to percent of total project cost for which ERR/FRR was calculated