Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Mx (apl I) Tertiary Educ Student Ass


  
1. Project Data:   
ICR Review Date Posted:
03/20/2013   
Country:
Mexico
PROJ ID:
P085593
Appraisal
Actual
Project Name:
Mx (apl I) Tertiary Educ Student Ass
Project Costs(US $M)
 300.00  290.30
L/C Number:
L7346, L7349
Loan/Credit (US $M)
 180.00  171.00
Sector Board:
Education
Cofinancing (US $M)
 -  -
Cofinanciers:
Board Approval Date
  12/13/2005
 
 
Closing Date
03/01/2010 12/31/2011
Sector(s):
Tertiary education (80%), Sub-national government administration (10%), Central government administration (5%), Secondary education (5%)
Theme(s):
Education for the knowledge economy (33% - P) Indigenous peoples (17% - S) Export development and competitiveness (17% - S) Administrative and civil service reform (17% - S) Poverty strategy analysis and monitoring (16% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
David W. Berk
Judyth L. Twigg Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:
The program (over 10 years in three overlapping phases) and the project (financed by the first of three planned APLs) had identical objectives as follows.

According to the Loan Agreement (p. 20), the objective was to “support the Government strategy to foster the sustainable and equitable and efficient expansion of tertiary education through student assistance through the development of a coherent student assistance system consisting of student loan and grant programs, compensatory interventions for disadvantaged students, and supporting the national tertiary education policy."

The PAD contains two differing versions of the PDO. On page 6, “the proposed APL would support the Government’s tertiary education strategy by strengthening the Government’s student assistance program to foster the sustainable, equitable and efficient expansion of tertiary education.” However, on pages vi and 8, "the proposed program supports the Government strategy to foster the sustainable and equitable expansion of tertiary education through student assistance.”

In assessing the project objectives, this ICR Review follows the Loan Agreement by using a four-part PDO: “fostering the sustainable, equitable and efficient expansion of tertiary education.” This broader set (including efficiency) seems to have been the original intention: there are related project components or actions; the restructuring (see next paragraph) paper repeats the broader set; and the ICR includes assessment of efficiency gains through the project.

Those portions of the Loan Agreement and PAD statements of the PDO that followed the words "through" and "by" are not objectives, but means to reach the objectives, namely outputs and inputs. As a consequence, the changes during project implementation - dropping the student loan component - were not changes in the PDO.

Key associated outcome targets for the end of the three-APL series were:

1. Increased numbers of graduates from tertiary education (expansion )
2. Increased share of enrollments for students from households in the two lowest income quintiles (equity objective)

However, there were no key outcome targets measuring the sustainability and efficiency objectives.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 03/01/2010

c. Components:
1. Support for the National Scholarship Program for Higher Education (PRONABES), a federally-designed program in which states participate voluntarily and share the financing 50-50 with the federal government; and for development of a national regulatory framework for tertiary education student assistance (appraisal estimate US$281.23 million, actual US$284.09 million): (i) Scholarships for an estimated 400,000 tertiary students from poor families, with the Bank financing 58% of the federal share or 29% of the total scholarship budget. (ii) Building federal regulatory and supervisory capacity to build and oversee an efficient and financially sustainable tertiary education system.

2. Analysis and institutional strengthening (appraisal estimate US$1.73 million, actual US$0.76 million): (i) Creating a database on the labor market performance of tertiary graduates and institutions. (ii) Studies providing the analytical foundation for strengthening tertiary education policy: income-contingent reimbursable grants, financing alternatives, equity, quality assurance mechanisms, links to the productive sector. (iii) Evaluation of student assistance programs.

3. Support for disadvantaged students (appraisal estimate US$5.92 million, actual US$4.03 million): (i) Early interventions to assist talented low-income students in upper secondary education (pilot). (ii) Expansion of the Program for Assistance to Indigenous Students in Tertiary Education Institutions (PAEIIES), which was managed by the National Association of Universities and Higher Education Institutions (ANUIES), in accordance with the project’s Indigenous Peoples Development Plan.

4. Support for state-level student loan programs (appraisal estimate US$10.58 million, actual $0): Program in one state (expected to be Quintana Roo) for loans and institutional strengthening. This was to be financed by a separate Bank loan for the project to the National Bank of Public Services and Works (BANOBRAS).

5. Promotion of private investments in student loans (appraisal estimate US$0.12 million, actual US$0): Design of ways to attract private capital and corresponding institutional arrangements, to be based on study of alternative financing for higher education, with implementation in later phases of the program.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Costs: Total project cost for phase I was originally US$300 million but diminished by US$10 million. The main reason was that component 4 for student loans and component 5 for private participation were removed from the project. Also, the Government implemented component 2(i) with other funding, achieving the project objective. Funding from an alternative source compensated for a late start to programs for indigenous students. Savings in components 2 (i), 3 (i) and 3 (ii) were reallocated to provide additional scholarships.

Financing: The original Bank contribution to phase I was to be US$180 million through two loans. The US$171 million Bank loan to the Government was fully disbursed. The second Bank loan, for US$9 million to BANOBRAS for student loans, was withdrawn 18 months after Board approval. BANOBRAS hardened its proposed terms for on-lending the Bank loan, and BANOBRAS and the newly elected authorities in the state of Quintana Roo never reached final agreement on legal and financial terms. No other state with implementation capacity expressed sufficient interest in such a program within the allotted time frame. The advent of the financial crisis may have contributed significantly to reduced interest in loan programs. The second APL was planned to be all-loan, and the third one a mixture of loans and some grants. Given its loss of interest in student loans, the Government decided not to proceed with the second and third proposed APLs.

Borrower Contribution: The Borrower’s original contribution to project costs was to be US$120 million. Its actual disbursement was US$119.3 million.

Dates: A slow start to the project under a newly elected government, and delays in initiating component 3(ii) because of coordination problems between the Government and the umbrella organization involved, required postponing the original closing date of March 1, 2010 by 22 months to December 31, 2011, leading to a project duration of six years.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High.

At approval the Government’s national development plan and education sector strategy both covered the period 2001-06. With the advent of a new administration in December 2006 and the financial crisis in 2008, its immediate priorities were restated under a Ten Point Program.

Keeping the expansion of tertiary education sustainable and efficient - and therefore not “budget-busting” - remains highly important. Making it more equitable was central to the Government’s national development plan’s “equality of opportunities” pillar, and to its education sector strategy that emphasized increase in opportunities to reduce inequality.

The Bank's Country Partnership Strategy for FY08-13 focuses on accelerating growth; improving competitiveness; promoting social inclusion and reducing poverty; developing infrastructure and assuring energy security and environmental sustainability; and strengthening institutions. The program’s and project’s focus on sustainable, equitable, and efficient expansion of tertiary education contributed particularly to the competitiveness objective and that of reducing poverty.

The sustainable, equitable, and efficient expansion of tertiary education was highly relevant to Mexico’s long-term development prospects, in particular its international competitiveness, and also to social inclusion and poverty reduction.

b. Relevance of Design:
Substantial.


The project’s causal chain to achieve the objectives was:

Expansion would result from an increased number of scholarships, student loans, and increased “terminal efficiency” (reduced rates of dropout and repetition) in tertiary education. The three aspects of expansion targeted by the program and project would have the following results chains. Sustainability would result from student loans. Equity would result from the targeting of scholarships and from measures to support indigenous students. Efficiency would result from labor market information on successful institutions to attract applicants and from increased “terminal efficiency” (on-time graduation) in tertiary education.

Given that the ERR of the project (Section 5 below) depended on this increased "terminal efficiency," i.e. on reduced repetition and dropout rates of students, there should have been additional (non-financial) support to all PRONABES students under the project, not just to indigenous students.

The omission of student loans during project implementation adversely affected the likely sustainability. It thus weakened the rating somewhat, but not enough to downgrade.


4. Achievement of Objectives (Efficacy) :

This review rates separately the expansion objective, as well as three aspects of that expansion: sustainability, equity, and efficiency.

The attribution of results in this section is solely to the project, since the project funded about one-quarter (and directly induced the funding of another quarter) of PRONABES’s expansion.

Foster the expansion of tertiary education: Substantial


Outputs:

Total tertiary enrollments were 2.2 million in 2003. They increased by about 700,000, or about one-third, over the project life (information from project team). The contribution of the project cannot be derived directly from the ICR.

PRONABES-financed students were 6% of the total in 2003. The number of students with scholarships provided by PRONABES increased from 161,787 in 2005 to 315,352 in 2011 (information from project team). The PRONABES increase was equivalent to about 20% of the increase in total enrollments. The total scholarship-years financed over the project life were about 1.6 million. About one-quarter were financed by the project as legally defined (Bank-plus-federal funding), and another quarter by states that were required to match the level of Bank-plus-federal funding under the project. The other half of PRONABES funding was provided by unanticipated major increases in both federal and state budgets for PRONABES scholarships. PRONABES’s total budget (federal plus states) went from MX$1.67 billion in 2005 to MX$3.14 billion in 2011.

The remaining 80% of the overall tertiary enrollment increase appears to have been financed almost entirely by an unanticipated doubling of federal and state budgets for tertiary education between 2006 and 2011 (from MX$43.2 billion to MX$80.9 billion).

Outcomes:

The number of tertiary graduates increased from 359,635 in 2004-2005 to 475,584 in 2010-2011, effectively meeting the PAD’s intermediate indicator target of 478,000 for that year.

Project (APL I) funding made a significant contribution to this increase as shown above. The planned contributions from the expected APL II and III by that date toward achieving the target did not materialize. But the institutional strengthening of PRONABES under the project (see next section) gave federal and state governments the confidence to channel double the amount of project (APL I) funding through PRONABES in this period. It is hard to imagine that the actual national graduation results would have been achieved without the expansion of PRONABES, which in turn was to a considerable extent attributable to the existence of the project.

Foster sustainability in tertiary education: Substantial

Outputs:

Tertiary education enjoyed strong political and social support during the project period. This commitment translated into a doubling of federal and state budgets for tertiary education and within them for PRONABES, as detailed above. PRONABES was one of the few programs that saw a funding increase during the financial crisis. These increases have been sustained up to the present. Through the years, more academic institutions have joined the program.

On the other hand, interest in Mexico in student loans was extremely limited during this period. The planned US$9 million in student loans under the project did not materialize. The economic downturn in 2008-2009 further reduced interest.

The project supported considerable institutional strengthening, in particular increases in the political and financial sustainability of PRONABES, through support to its Annual Meetings and -- most importantly -- to development of its financial system and management information system. These furthered communication and compliance between the federal and state governments, and enabled external dissemination of evaluations of PRONABES. The project also supported the sustainability of the Academic Support Units for indigenous students at participating institutions.

Outcomes:

The increases in tertiary education budgets appear to reflect a strong social demand and consensus in favor of government spending on tertiary education in general. Such spending can be expected to be sustained in the future.

The improvements under the project and the results have made PRONABES a more credible implementing agency for government policy. PRONABES in its strengthened form represents a major resource for tertiary education promotion in the future.

For purposes of this ICR review, the social and governmental support for tertiary education spending outweighs the abandonment of the student loan component under the project and APL II and III.

In the government’s plan for continuing operation of the project, it has committed to continue promoting increased financing to sustain and expand PRONABES. The current education strategy mentions the government’s intention to increase the coverage of PRONABES in all tertiary education subsystems. Seventeen of 24 universities that participated in support for indigenous students (PAIIES) have indicated that they will maintain the program with their own resources; 12 of them have agreed to provide at least 70% of the financing received under the project. This may be optimistic. However, any shortfall would only have a limited impact on overall sustainability.

The fact remains that the long-term financial sustainability of the tertiary education system will require some contributions from beneficiaries, as originally proposed under the project and program. It is encouraging therefore that the government of Mexico continues to realize this, and has recently begun tentative exploration of some options in this area. But interest in student loans declined considerably during the financial crisis. There is no guarantee that a new contribution scheme will materialize any time soon.

Foster equity in tertiary education: Substantial

Outputs

The promotion of greater equity was an important plank of national and education sector plans and strategies.

The primary target group under the project was economically and socially disadvantaged tertiary education students.

The annual numbers of PRONABES scholarships increased considerably as detailed above. Targeting, with supporting rules and the use of management information systems, was sharpened progressively. 100% of beneficiaries came from households with income below four times the minimum wage by project closing, versus 96% at the start, and this also meant that 100% of them were from the two lowest income quintiles.

Starting in 2007, priority has been given very successfully to former recipients of the Oportunidades conditional cash transfer program for lower-income families and to indigenous students. For the indigenous students program, there was a substantial delay in establishing the contract between GOM and the university association ANUIES that manages the program. As a result, Bank funds arrived only after a delay of three years. However, unanticipated continuing Ford Foundation (US) assistance permitted reaching 9,600 students by then. The project then supported existing students and added almost 2,400 more, so 11,918 were assisted instead of the originally envisaged 3,500.

Outcomes:

Key outcome target #2 was for an increase in the share of national tertiary enrollments accounted for by students from households in the two lowest income quintiles. There was an increase from the baseline share of 10% to the 2010-2011 share of 20.6%, meeting the informally revised target of 20%.

For ex-Oportunidades and indigenous students there were intermediate results indicators, with the following outcomes (% of PRONABES beneficiaries):

2004-2005 Target 2010-2011

Ex-Oportunidades 8.7 10 17.9
Indigenous 6 10 12.6


While there was a Government policy in favor of increased equity in tertiary education, which would likely have affected PRONABES’s allocations in this period even in the absence of the project, the project resulted in better information and most likely in farther and faster shifts in allocations than would have occurred without it.

The policy of equity promotion is likely to continue in future, Any shortfall in funding the indigenous student support program could produce shifts within the poor student population rather than reductions.

Foster Efficiency in Tertiary Education: Modest

Outputs:

The PAD quoted analysis showing that the private return to tertiary education in Mexico as of the early 2000’s ranged from 16%, taking account only of earnings foregone while studying, to 10% including also private direct costs. The social return including also public direct costs was 7%, which is acceptable. These estimates did not take into account other benefits (positive externalities) from investment in tertiary education, such as (according to the PAD) improved technology absorption. These rates of return would be improved if repetition and dropout rates diminished as intended under the project.

Increasing the proportion of students who stay in tertiary education and graduate was an important objective and benefit of the project. However, there was inadequate provision in the project activities to encourage and enable students to achieve this. For the majority of students who were not indigenous, the only measure was that achieving a satisfactory grade point average was a condition of renewal of their scholarship each year. Indigenous students did receive non-financial assistance through the PAEIIES program.

Data collection methods and the six-year project life have not yet permitted PRONABES to collect data on the “terminal efficiency” (proportion of a student cohort that graduates in five years). Instead, the year-to-year retention rate has been used.

Outcomes:

There was a modest increase during the project life in the efficiency with which the tertiary education system as a whole turned numbers enrolled into numbers of graduates. The tertiary graduates-to-enrollments ratio in a given year increased from 13.5% in 2003 to 16.5% in 2011. The project contributed to this outcome.

Under the project, the proxy for “terminal efficiency" -- the year-to-year retention rate -- was estimated at 65.3% in 2003-2004, and the target for 2010-2011 was 75%. The actual figure was 87.3% in 2010-2011, a considerable increase. Nevertheless, this figure implies that approximately 57% of the cohort of PRONABES students who started in 2010-2011 will complete their studies in 2014-2015, still an undesirably low proportion. The increase in the retention rate was not enough to lift the economic rate of return to a satisfactory level (see Section 5).

Indigenous students participating in PAEIIES also improved their year-to-year retention rates, though by less than was targeted.

PRONABES increased its efficiency during the project period and kept its administrative costs at less than 1% of the total scholarship budget, through a new financial system and strengthened management information system, online applications, reduced paperwork for informal sector and rural students, and arrangements to disseminate information and coordinate activities with the Oportunidades program.

Only the improvements by indigenous students would probably have occurred without the project.



Progress Toward Objectives of Entire APL Program

The program (APL I-III) and its first phase (APL I – the project) had the same objectives, so that progress under the project represents progress towards the program objectives. Generally speaking, progress was satisfactory, except as regards student loans, which were dropped from the project, and which contributed to the Government’s decision not to pursue APL II and III. This affected the long-term sustainability of the tertiary education system. However, more encouragingly, the government has recently begun again to explore options in the area of beneficiary contributions.


5. Efficiency:

a. Economic Rate of Return:


The PAD's economic analysis assumed that all project benefits came from increasing the “terminal efficiency” of PRONABES beneficiaries, defined as the proportion of a cohort of students that graduates in five years. This increase produces more tertiary graduates (than without the project) who benefit from increased lifetime earnings compared to secondary graduates. The PAD's sensitivity analysis showed how much this proportion of "on-time" graduates must increase to produce a certain economic rate of return, but the PAD did not estimate a single ex ante ERR for the project based on this analysis.

The ICR (pp. 32-3) reports a “terminal efficiency” achieved of 63% (based on the latest available figures, which are for year-to-year retentions), hence a project ERR on the PAD basis of between 3.6 and 4.5%. It states that this justified the project from an efficient use of capital perspective. This review disagrees, considering this an inadequate economic rate of return.

The ICR states that other benefits should also be taken into account, in particular the values of increasing equity and of improving project administration. It makes the following analysis. A proportion of PRONABES graduates and their eventual families can be expected to escape poverty, with substantial savings to future government budgets for anti-poverty programs (in this calculation, programs of conditional cash transfers and free health insurance); the ICR assumes that 50% of graduates/families so escape (information from TTL). The ICR does not provide the detailed calculation, but states that adding these future government savings to the benefits in the PAD analysis yields a final project ERR in the ICR of 8.3 to 16.4%, which is acceptable to very good.

However, this approach is not acceptable. Education projects evaluated by IEG only include benefits that are directly linked to the objective and activity. Also, the expected government savings on other anti-poverty programs may not materialize. While all PRONABES students came from households in the bottom two income quintiles, not all were eligible for other government anti-poverty programs; for example, only 18% were from households benefiting from the Oportunidades program. The programs do not necessarily reach all eligible households now, so removing some households might not reduce total expenditures. Also, secondary graduates are also not generally poor, and some who are poor will continue to escape in the without-project case. Finally, projects in other sectors also raise beneficiary incomes, but the type of positive externality cited in the ICR for this project (savings on future government anti-poverty programs) is not included in their ERR calculations.

There were two other project achievements in the area of efficiency. PRONABES kept its costs below 1% of its scholarship portfolio, an excellent level. It also simplified the application process, saving students and their families considerable time and effort.

This ICR review considers that the inadequate economic rate of return outweighs project achievements in the area of value for money.

Efficiency Rating: Modest

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
Yes
8.3%
100%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Based on High relevance of objectives; Substantial relevance of design; achievement of the tertiary education objectives that was Substantial as regards expansion, Substantial as regards sustainability, Substantial as regards equity, and Modest as regards efficiency; and Modest project efficiency, the outcome based on the project development objectives is rated Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

There is strong social demand for tertiary education. There is also strong support at both the federal and the state level for PRONABES. Its federal budgets have been rising strongly, and states have made matching amounts available, despite some macroeconomic difficulties. Public dissemination of positive evaluations of the program is expected to help maintain support. PRONABES has substantially increased its capacity through hiring specialized staff, has improved its financial management and efficiency, and has refined the targeting of its grants.

There could be some difficulty maintaining budget allocations for the program assisting indigenous students, which would pose some risk to the development outcome. However, seventeen of 24 universities that participated in support for indigenous students (PAIIES) have indicated that they will maintain the program with their own resources, and 12 of them have agreed to provide at least 70% of the financing received under the project. This may be optimistic, but any shortfall would have only a limited impact on overall sustainability.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:
Project preparation, design, and quality at entry had a number of strengths. It identified the need for increased funding and improved targeting of scholarships, leading to sound design of the PRONABES component. It was based on lessons from international experience. It included an innovative Indigenous Peoples Plan, involving the identification of a functioning program for indigenous students (PAIIES) for support and expansion. The rationale for an APL lending instrument was strong, providing support for a reform program that was long-term and required piloting. Finally, intermediate results indicators were included, which served as a precaution and basis for eventual assessment in case the three APLs planned did not materialize.


However, there were also a number of shortcomings. The case for the project in the PAD was based on the difficulty of expanding public expenditures on tertiary education, which completely misread the situation, rather than on the Government’s and the Bank’s interest in demand side financing, and on the need to strengthen PRONABES as the implementing agency. Inconsistencies about project objectives (i.e. was efficiency included or not?) showed up within the PAD and between the PAD and the Loan Agreement. The PDO as drafted included both objectives and means to achieve them. The project was complex in terms of the numbers of entities involved and of activities. The project management arrangements gave PRONABES more responsibilities than its authority vis-a-vis other project entities, and there was no governance structure (e.g. steering committee) to resolve problems. Also, PRONABES initial capacity was insufficient.

Legal agreements, contracts, and arrangements to secure local budget funding were not finalized before Board approval, leading to effectiveness and implementation delays. As for the results framework, there were problems with the choice of key performance indicators, baseline figures, targets, and ability to monitor indicators (see Section 10 below). Finally, the PAD’s economic analysis derived all benefits from increased on-time graduation rates, but only indigenous students (6-13% of total numbers) received related support within the project.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
The Bank provided active supervision, including attempting to get the Borrower to solve the above-mentioned quality-at-entry problems of this project. For example, the project team consistently stressed the need to resolve the legal and administrative complexities holding up the transfer of resources from GOM to ANUIES for the indigenous student support program. The Bank also made a special effort in financial management supervision through training and an action plan to resolve problems in this area (see Section 9 below). It moved quickly to downgrade the implementation progress rating of the project.

However, the way the PDOs were drafted caused recourse to Board approval of the project restructuring when the student loan component was dropped. Also, at restructuring the project's implementation arrangements should have been strengthened. The Bank team did not achieve the revision of performance indicators at restructuring by overcoming Government opposition to changes. This made it difficult to assess achievements under the project. Also, the ICR (page 24) acknowledges Borrower comments that there were delays in some Bank procurement approvals and in disbursement of the final loan amount, by stating that these processes could have benefited from greater (Bank) efficiency across the board.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The Government provided rapid increases (roughly a doubling) in budgets for scholarships at both federal and state levels. It thus fulfilled its commitment of total funds to the project and did much more. It achieved a significant expansion of graduate numbers in tertiary education.

On the other hand, the Government failed to put in place governance arrangements (e.g. a steering committee) to secure project actions by agencies outside PRONABES, which did not have the authority to compel their performance, and to provide enough capacity for the National Coordination [Unit] of PRONABES (CNPRONABES) to undertake the management of the project. Also, it delayed provision of funds for certain components or actions, such that the PRONABES management information system and the component for indigenous students were delayed by as much as three years. Further, it failed to bring about implementation of the loan component, sacrificing Bank funding of US$9 million here and two planned APLs adding to US$390 million, and setting back the development of a more sustainable system of support for the long-term expansion of tertiary education. At restructuring, it did not agree to formally revise targets, even those that clearly required this (in particular the equity PDO measure - new data showed that the end-project target had been achieved by project start! - and other mainly intermediate indicators). This was perhaps out of fear of criticism from the domestic political opposition, media, and education advocates that its performance was falling short.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
PRONABES executed its rapidly increasing budgets in a timely fashion. It allocated scholarships in line with the targeted distribution, and reached or exceeded a number of targets, especially those reflecting the equity objective. It greatly improved both its staffing and its systems, especially its financial management system, and its internal efficiency, thus keeping its overall costs very low.

On the other hand, PRONABES badly mismanaged a Japanese PHRD preparatory grant (see Section 11b). It should not have undertaken project management with its minimal initial capacity (only two staff in CNPRONABES during project preparation). It could have been more proactive in getting certain project components underway, both legally (indigenous student program) and financially (seeking budget authority earlier). There were some procurement delays. PRONABES performed relatively poorly in financial management. Throughout project implementation, problems included slow submission of disbursement requests, delayed financial management reports, and the need for reclassification of some statements of expenditure. Some audits of the use of loan proceeds were qualified. The auditors were particularly strict on this project, presumably because of prior mismanagement, and some of the infractions were minor (according to the Project Team). The shortcomings did not involve substantive financial mismanagement.

The National Bank of Public Services and Works (BANOBRAS) performed well in its role as fiduciary agent for the Bank loan. It could have communicated better with states over the student loan component, although other factors beyond its control contributed to that negative outcome.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The M&E design (PAD Annex 3) may be considered along two dimensions: what was to be monitored, and who would do it and how.


As for the results framework, there were problems with the choice of key performance indicators, baseline figures, targets, and ability to monitor indicators.

The two key outcome targets measured the overall expansion objective (number of tertiary graduates) and the equity objective (share of enrollments from the two lowest income quintiles). However, the overall expansion target was beyond the capacity of the project to deliver. Also, there were no key outcome targets measuring the sustainability and efficiency objectives.

Given that three APLs were envisaged, the key outcome targets were for the end of the third APL in 2013, although some interim values for earlier years were also defined in the PAD. In addition, there were a number of intermediate results indicators in the PAD: one for sustainability, four for equity, and five for the efficiency objective. These were generally for 2009, when the overlapping APL I and planned APL II were both scheduled to end. Most of the indicators were relevant and useful for attribution and potentially for project management, but the “terminal efficiency” indicator that was key to project returns was particularly problematic (see section 10b).

PRONABES was responsible for most of the indicators; the National Association of Universities and Higher Education Institutions (ANUIES) for the indicators for indigenous students; the state-level institutions for student loans; and the federal Ministry of Education for sectoral data. The data were to come from PRONABES surveys and from the databases of the other institutions.

b. M&E Implementation:
Some baselines and target numbers in the PAD had inconsistencies. Some indicators were hard to measure at project start due to lack of capacity in PRONABES. Despite its importance, “terminal efficiency” remained hard to measure as it required tracking students for five years, and so annual retention rates and graduations were measured in the interim.

Both PRONABES and ANUIES received related project funds with long delays (up to three years), and it took a long time before their management information systems and data reporting gradually improved. On the positive side, there were five evaluations of the PRONABES program, published on its external website, and two impact evaluations are currently underway.

During the mid-term review in June 2009, PRONABES calculated revised baselines and targets for some indicators. Some revisions were far-reaching. However, the revisions were not included in the March 2010 restructuring. The ICR (p. 12) states that this was because "further revisions to the MIS were being made." In fact, the Bank team was not able to overcome government unwillingness to change the indicators formally, possibly out of its fear of domestic criticism for lack of performance (information from the Project Team). Some baselines and targets were again informally revised and agreed with the Bank team in 2011. Finally, the ICR team constructed a set of indicators for the ICR, building on the PAD’s intermediate results indicators that measured outcomes directly attributable to project activities.

The failure to revise indicators as part of the 2010 restructuring, when it was clear there would only be one APL and not three, and that it would end in 2011 rather than 2009, was a serious omission by the Borrower and the Bank. It greatly complicates reaching judgments on project performance and accountability for results.

a. M&E Utilization:
Data generated by project systems was used in several evaluations of the program. Limited use was also made of the data on students to refine PRONABES eligibility criteria to sharpen targeting.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
The sole safeguard policy triggered by the project was OP/BP 4.10, Indigenous Peoples. This was triggered because indigenous students faced problems of limited information, lower income, and poor academic preparation that led to their very low enrollment share in tertiary education. Under the project, the Indigenous Peoples Development Plan targeted financial assistance to them. PRONABES committed to provide a scholarship to all indigenous tertiary students supported through the non-financial PAEIIES program, which the Bank loan also financed. (PAEIIES is the Program for Assistance to Indigenous Students in Tertiary Education Institutions.) PRONABES exceeded its target for the share of indigenous students in total beneficiaries.

b. Fiduciary Compliance:
PRONABES mismanaged contractual procedures under a Japanese PHRD preparatory grant; some contracts were reissued but repayments were not required. The director at the time was fined and temporarily suspended. On the other hand, there was no dishonesty, and the money was well spent on plans and programs. Auditor criticism was serious, and the Project Team added that thereafter the auditors were especially demanding with this project. The limited procurement conducted under the project was done in accordance with Bank policies, but delays were reported. There were persistent financial management problems throughout project implementation despite Bank training: slow submission of disbursement requests, delayed financial management reports, the need for reclassification of some statements of expenditures, and qualified audits of use of loan proceeds. An action plan was finally implemented soon after restructuring and eventually resolved the issues including audit qualifications. The Bank is reviewing the final audit report, which has been received and is unqualified; use of funds has been reasonable and disbursements well documented, but some problems remained with allocation of expenditures among components and sub-components and disbursement categories.

c. Unintended Impacts (positive or negative):
None reported in the ICR.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Negligible to Low
Political and financial support for PRONABES is high, and important institutional capacity was built under the project. A majority of universities that received project support for indigenous students have indicated that they will maintain the program using their own resources. 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
IEG offers the following lessons:

Bank understanding of political and financial context is critical for judging country commitment and the case for Bank funds. In this case, the PAD's assessment of Mexico’s inability to spend more on tertiary education was completely belied by a doubling of the related budgets during the project period, while interest in student loan programs proved much less than expected.

PDOs should not mingle objectives and means (outputs and inputs) to reach them. In this case, there were no changes to the PDOs during project implementation but only to the means, and Board approval of a restructuring was obtained unnecessarily.

Project design should address the key factors determining the project's ERR, requiring pro forma economic analysis upstream. In this case, additional support measures to students, to reduce repetition and dropout rates, could perhaps have raised the ERR to a satisfactory level.

Other lessons, taken from the ICR but paraphrased, are:

Objectives and targets that are set carefully, baselines that are established before project approval, and monitoring that is within the capacity of the agency responsible are required for successful and useful M&E. In this case, the project had numerous problems with baselines, targets and monitoring.

Giving the implementing agency responsibilities that are beyond its authority and/or capacity will result in poor performance. In this case, PRONABES could not require other units of the Ministry of Education and institutions outside it to perform their project roles satisfactorily, and therefore it needed to increase substantially its own initial capacity.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR provides a clear history of the key outcome targets and results indicators under the project and their revisions during implementation. It covers the great majority of the available evidence on project developments and outcomes (the Project Team subsequently provided useful supplementary information separately).

The quality of analysis is satisfactory. The ICR is clear on inconsistencies in the statements of the project development objectives in various documents. It has a good discussion on quality at entry issues. It also has a good analysis of project outcomes by objective. However, it does not criticize the case for the project made in the PAD. It does not deal with the mingling of objectives and means in the PDO, which led to an unnecessary recourse to Board approval of a restructuring. It is coy on the real reason the key performance indicators were not revised as needed at the formal restructuring. It does not analyze the attribution of results to the project or provide a “counterfactual” situation. Its assessment of risk to the project’s development outcome would be strengthened by including Government and university pledges for the future. It has made an extensive and original economic analysis, but its judgment that the conventional ERR of the project was adequate is questionable, while its inclusion of certain externalities that greatly increase the ERR is not acceptable.

The ICR provides a pertinent set of lessons based on specific experiences under the project. It is results-oriented and internally throughout, and consistent with guidelines. It is candid throughout, except on the non-revision of key performance indicators. It is generally concise throughout the implementation narrative, with the exception of some parts of the M&E discussion.


a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions