Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Pakistan: Punjab Education Sector Project


  
1. Project Data:   
ICR Review Date Posted:
10/21/2013   
Country:
Pakistan
PROJ ID:
P102608
Appraisal
Actual
Project Name:
Pakistan: Punjab Education Sector Project
Project Costs(US $M)
 3,350.02  4,364.11
L/C Number:
C4586
Loan/Credit (US $M)
 350.00  401.82
Sector Board:
Education
Cofinancing (US $M)
 0  17.56
Cofinanciers:
CIDA, DFID
Board Approval Date
  06/04/2009
 
 
Closing Date
06/30/2012 06/30/2012
Sector(s):
Primary education (39%), Secondary education (39%), Public administration- Education (22%)
Theme(s):
Education for all (60% - P) Public expenditure financial management and procurement (25% - S) Gender (15% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Susan Ann Caceres
Judyth L. Twigg Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Financing Agreement (p. 5), the objectives of the project were to "improve access and equity, and the quality and relevance of education in Punjab." The same objectives were noted in the Project Appraisal Document (pp. vi and 6).

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

There were two components (actual amounts noted reflect total project costs and are not broken down in the ICR by IDA contribution). Disbursements were made against a set of Eligible Expenditure Programs, which were conditioned on the achievement of Disbursement Linked Indicators (DLIs), when 80% of these indicators were achieved in each fiscal year. Eligible Expenditure Programs were teacher salaries, School Council capacity building, District capacity building, School Council recurrent budget, and monitoring (PAD, p. 55). DLIs were: preparation of Medium Term Fiscal Framework; improved quality of teaching; decrease in number of schools with missing facilities; increased transition from primary to middle school, middle to high, and high to higher secondary; continued delivery of free textbooks to public schools; timely provision of stipends to eligible girls enrolled in grades 6-10 in public schools; increase in public financial support for students in low cost private schools; enhanced accountability of teachers; increased capacity of School Councils for effective school management; reduced teacher absenteeism; teacher recruitment on a merit and need basis; transparent examination and assessment system; and good quality information on student learning (PAD, pp. 41-42).

1. Financing of the Punjab Education Sector Reform Program (PESRP) (appraisal, US$ 3,340.0 million of which IDA was to finance US$340.0 million; actual, US$4,361.7 million) contained four subcomponents:

    • Enhanced Education Sector Expenditures and Improved Fiduciary Environment was to focus on education sector expenditures and fiduciary reforms to promote improvements in the composition, quality, and efficiency of sector expenditures. The component was to support strengthening of financial management practices to instill a culture of remedial action on audits and build capacity to keep spending in line with sector objectives and reduce internal audits and audit observations. It also was to work to improve financial management at the provincial level so that timely expenditure reports were completed, and implement new rules and practices in procurement of goods and works (e.g. civil works, textbooks, and furniture). There were no Eligible Expenditure Programs for this subcomponent.
    • Improved Quality and Relevance of Education was to strengthen the system for measuring student learning. It was intended to improve quality by enhancing teacher professional development and teachers' instructional practices and use of learning materials. It was to design and pilot a provincial teacher accreditation system for teacher certification and licensing. Performance-based incentives were to be provided to well-performing schools, along with additional support to low-performing schools. The Eligible Expenditure Program for this component was the performance-linked incentives for schools.
    • Improved Access and Equity was to focus on: (1) decreasing the number of schools with missing facilities; (2) increasing the transition from primary to middle and secondary level; (3) providing demand-side interventions such as timely delivery of free textbooks in Grade 1-10, open competition in textbook publishing and printing, and stipends to eligible girls in Grades 6-10 in fifteen low-literacy districts; (4) increasing public financial support for the private sector to establish low-cost schools in areas with low enrollment through the Punjab Education Foundation; and (5) offering supply-side improvements such as filling of teacher vacancies, reducing missing facilities in schools, and adding grades to schools (i.e. middle and secondary) in rural areas. The Eligible Expenditure Programs for this subcomponent were the girls' stipend program and public-private foundation.
    • School Management, Governance & Monitoring was to create an enabling environment for schools to function more effectively by enhancing accountability of teachers to parents and stakeholders at the local level, and by increasing capacity of School Councils for effective school management to reduce teacher absenteeism. It also was to use merit criteria to recruit and place teachers and establish a transparent examination and assessment system to provide quality information on student learning. A streamlined mechanism was to be put in place to transfer recurrent non-salary budgets to all School Councils. Greater capacity for planning and evaluation at the provincial level was to be created by strengthening field and district education monitoring. The autonomous functions of the Punjab Examination Commission were to be strengthened to ensure better quality provincial assessment of students in key subjects. The Eligible Expenditure Programs for this subcomponent were teacher and staff salaries, School Council capacity building, School Council recurrent budget, and District-level capacity building and monitoring.

2. Technical Assistance (appraisal, US$ 10 million of which IDA was to finance US$10 million; actual, US$2.41 million) was to finance generic cost elements of the project through a system of Special Drawing Accounts. These accounts were to be held with the Program Monitoring and Implementation Unit for consultancy services in the areas of expenditure tracking, third party validation and monitoring, training and other capacity-building activities, education strategy development, and budget planning and management. This component was to support the medium-term education sector framework of the Punjab Education Sector Reform Program to strengthen financial management and procurement systems and processes in the sector. It was to fund the use of third-party validation for programs such as delivery of textbooks, teacher recruitment, and environmental enhancements. It was also to provide technical assistance to improve the quality of education, enhance access, and strengthen governance and monitoring of the performance of ongoing programs.

An Additional Financing Agreement was approved by the Board in February 2011. After the 2010 floods in the province, the Government requested support to maintain an uninterrupted flow of funds to the core education program, since there was a financing gap after the Government funded relief and reconstruction work. Two project covenants were dropped (establishment of a teacher accreditation and licensing body) because of lack of clarity in the role of federal accreditation versus the role of the provincial government.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Costs: Unused funds (SDR 5.1 million equivalent) were reallocated from the second to the first component, equally distributed across the 10 disbursement-linked indicators, which increased each indicator from US$11.33 million to US$11.59 million. One half of the amount allocated for the disbursement-linked indicator for upgrading of schools was cancelled (US$8.1 million), since the target related to upgrading schools from primary to middle level was not achieved. Project costs were 130% higher than estimated. The ICR does not explain this escalation in project costs.

Financing: The IDA credit for the Sector Wide Approach (SWAp) was estimated to be US$350 million, with US$65.70 million in parallel financing from the UK Department for International Development (DFID). In March 2011 US$50 million Additional Financing was approved of which US$ 48.5 million disbursed and the remaining amount was cancelled. The actual amount financed by IDA was US$ 401.82 million, with an estimated US$40.43 in parallel financing from DFID. The Canadian International Development Agency (CIDA) provided US$17.56 million in joint financing. DFID committed GBP 50 million for 5 years at appraisal. The period of support was shortened to 3.5 years, but the agency increased the total amount of the grant to GBP 80 million, linked to additional targets established by DFID related to missing facilities in five districts (the ICR does not provide the US dollar equivalents for these contribution amounts). In addition to its technical, financial, and advisory support, DFID also provided high-level advisory assistance to the provincial and district leadership, including a special advisor to the Chief Minister of Punjab.

Borrower Contribution: The Borrower provided US$3.9 million, 133% higher than what was estimated. The ICR does not provide an explanation for this higher-than-estimated contribution.

Dates: The project closed as planned without an extension. On June 6, 2012 there was a level two restructuring, which reallocated resources as noted above.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Relevance: Substantial
Even though Punjab is the most prosperous province in the country, school enrollment levels are low. At the time of preparation, only 62% of children were enrolled in primary school, with levels even lower for rural girls and children from poor households. Transition rates from primary to secondary were low, with Net Enrollment Rates respectively of 20% and 11%. Issues in the education sector at appraisal included: (1) insufficient resources allocated to education; (2) inadequate district-level capacity for planning, budgeting, and managing and implementing reforms; (3) limited parent and community oversight and participation in schools; (4) weak support systems for teachers with inadequate provision of materials and aids; (5) patronage-based public school teacher hiring; (6) lack of teacher standards and quality assurance in teacher education and professional development; and (7) inequitable provision of school facilities. All of these factors affected the public education system and caused poor outcomes in terms of access and learning, with independent assessments finding students performing significantly below grade level.

The Bank's current Country Partnership Strategy (FY2010-13 extended to FY14 with the Progress Report) emphasizes improving equitable access to quality education. It also notes that improving the efficiency of public expenditures is critical to achieving development objectives, since modest progress in human development indicators is a function of both inadequate investment and ineffective use of available resources (p. 14). The project's objectives were substantially relevant throughout the life of the project to the education sector's needs and to the Bank's country strategy.

b. Relevance of Design:

Design: Modest
To address challenges in the education sector, the Government of Punjab initiated a medium term Education Sector Reform Program in 2003. This Sector Wide Approach was designed to support this Program, specifically to improve governance and accountability in education service delivery and combine education reforms with fiscal and public finance management interventions to improve effectiveness and efficiency of public expenditures. Strengthening of monitoring and evaluation systems was intended to focus the Government's efforts during program implementation. Changes in governance and budgetary/financial management underpinned project design by revising structures, incentives, and practices through a results-based approach, which was also intended to promote teacher accountability. The interventions and DLIs were designed to improve service delivery performance, particularly at the school level. The project promoted regular administration of student assessments, with technical support provided by Development Partners. In response to the draft ICR review, the project team indicated that it views the objectives of improving education quality and improving relevance as the same, and therefore the strategies used to improve education quality (e.g. testing, mentoring of teachers, provision of textbooks, and incentives to low-performing schools) were applicable to improving the relevance of education. Disbursements were to be linked to the achievement of targets (at least 80% had to be met each fiscal year) and were made against a set of selected Eligible Expenditure Programs: teacher and sector staff salaries, girls' stipends, financing of low-cost private sector schools, capacity building for school councils and districts, and monitoring.

The ICR (p. 7) states that a Quality Enhancement Review endorsed design.

However, there were shortcomings. Most importantly, while the DLIs provided a strong results orientation and were linked to the objectives, some were poorly defined or not focused on the needed actions. For example, in the case of student assessment, the DLI did not relate to actions to demonstrate improvements in test design, administration, and analysis. Finally, the ICR reports that the lack of detail in the Technical Assistance Component was a design weakness.


4. Achievement of Objectives (Efficacy) :

Punjab was affected by large-scale floods and heavy rains in 2010 that impacted the functioning and performance of the education sector. The floods caused extensive damage to school infrastructure, loss of school and student resources, disruption to school operations, economic losses due to the physical displacement of households, and the diversion of Government attention, efforts, and scarce funds to disaster relief, recovery, and reconstruction activities. The floods most likely adversely impacted the achievement of the project's development objectives.

Improve access: Modest

Outputs:

The project supported low-cost private schools where enrollment increased from approximately 500,000 students to more than 800,000 by the end of the project. This program was expanded from 18 to 29 districts. The ICR (p. 20) also reports that the funding of the program was based on students attaining a minimum level of learning to create incentives for the private schools to be responsive to the needs of students and ensure quality education, but no additional information is provided to be able to gauge the effect of the program. The ICR does not describe whether this program crowded out existing private or public schools.

972 primary schools were upgraded to middle schools, which did not meet the target (1400 schools). 578 middle schools were upgraded to high schools, which exceeded the target (400 schools). School Departments prepared an implementation plan and developed a priority list of 1,400 primary schools and 400 middle schools that were identified for upgrading, which gave priority to girls and rural areas (ICR p. 39). Third party validation showed non-compliance with the priority list of schools, as only 44 of 682 schools were on the list (ICR, p. v).

Outcomes:

The primary net enrollment slightly increased from 61.7% in 2007 and 63.6% in 2012, not meeting the downwardly revised target (65.0%). 2011/2012 data were not available at the time of the ICR. See Table 1 for trend data. DFID implemented a household survey in 2011 showing that enrollment had improved, but the extent of this improvement is not provided in the ICR. The ICR provides several explanations for the stagnation in primary enrollments: (1) the floods could have dampened demand for school; (2) the decelerating economy and decreasing commodity prices affected the Government's ability to expand investments in education in keeping with a growing school-age population; (3) delays in implementation of provision of missing school facilities. The combination of the floods and the economic slowdown could have made it harder for poor households to send their children to school; the estimated out-of-pocket monthly expenditure to send a child to public school in Sindh province was PKR 600 (approximately US$6.11), and most families have 3-4 school-age children.

The middle-level net enrollment rate slightly increased from 20.1% in 2007 to 25.4% in 2012, meeting the upwardly revised target (24%). See Table 1 for trend data. Progress could be related to the project interventions to enhance participation at the middle school level (stipends for secondary school girls, school upgrades, and provision of missing school facilities).

The primary completion rate slightly increased from 50.5% in 2007 to 53.5% in 2012, not meeting the target (55%). Household survey data were used to calculate this rate, and therefore it includes both government and private schools. 2011/2012 household survey data were not available at the time of the ICR. The ICR reports that its interventions to improve quality (merit placement of teachers, capacity building of school councils) promoted greater retention. The ICR (p. 17) also reports that the children who entered school during project implementation have yet to complete the primary school cycle, and so the full effect of the intervention is yet to be realized.

Table 1
IndicatorBaseline (FY07)
(%)
FY08
(%)
FY09
(%)
FY11
(%)
FY12
(%)
Target
(%)
Primary net enrollment rate61.760.561.260.763.665.0
Middle level net enrollment rate20.119.021.822.625.424.0
Primary completion rate50.551.752.753.653.555

PSLM data for 2011-12 were provided by the region after reading IEG's draft review, which were not available at the time of the writing of the ICR.


Improve equity: Modest

Outputs:

The public-private partnership program provided low-cost schools in some hard-to-reach disadvantaged communities. 200 schools were supported under the New School Program. The ICR (p. vii) notes that there were delays in transfer of funds to the Punjab Education Foundation during the project, which created delays in implementation.

The project supported female students by recruiting more female teachers (no numbers provided) and giving female teachers better access to training (no numbers or impact provided).

A Girls Stipend program was implemented, and in FY2011-2012 the program covered 385,000 girls in 16 districts. This program targeted the worst-off and underserved areas of the province (ICR, p. 11). Based on third party validation, 93% of eligible girls received the stipends each year. The delivery of the stipends took more than five months, while it was planned to take less than 3 months (ICR, p. 29).

Outcomes:

The female primary net enrollment rate increased from 59% at baseline to 62.2% in FY12. See Table 2 for trend data. The ICR does not provide middle level net enrollment data disaggregated by gender.

During the life of the project, the ratio of female-male net enrollment for both primary and middle education in rural Punjab increased respectively from 89% in 2006/2007 to 93% in 2011/2012 (which met the target of 92%) and from 74% in 2006/2007 to 78% in 2011/2012 (which did not meet the target of 86%). See Table 2 for trend data. However, the ICR indicates that ratio-based indicators should be avoided, since the estimator is sensitive to slight changes in the numerator and the denominator. Changes in the indicators between 2007-2008 were related to changes in male enrollment, as the ICR suggests that boys withdrew because of the higher forgone earnings relative to girls.

Over the project period, the growth rate in female middle-level net enrollment (rural) was almost twice that for males (4% and 2%, respectively). The female primary growth rate is not provided, and the ICR does not provide middle-level data disaggregated by gender to be able to compare the trends for both boys and girls to determine if the growth rate for girls was sufficient, given their lower enrollment.

Table 2
IndicatorBaseline (FY07)FY08FY09FY11FY12Target
Male primary net enrollment rate63.662.363.462.465.065% (overall)
Female primary net enrollment rate59.458.858.759.062.265% (overall)
Female-male ratio net primary enrollment, rural89.791.289.392.593.592.0
Female-male ratio net middle enrollment rate, rural74.591.682.482.778.286.0

PSLM data for 2011-12 were provided by the region after reading IEG's draft review, which were not available at the time of the writing of the ICR.

Findings from the evaluation of the girls' stipend program in Punjab showed a positive impact of the program on female enrollment. Beneficiary adolescent girls were more likely to progress through and complete middle school; girls eligible for benefits increased their rates of matriculation and completion of high school; there was suggestive evidence that participating girls delayed marriage and had fewer births by the time they reached 19 years of age; and positive trends in enrollment were also observed for boys in these districts. The delivery of the stipends took more than five months, while it was planned to take less than 3 months (ICR, p. 29). The ICR (p. 17) reports that girls' secondary enrollments may have been protected by the stipend program, as secondary male enrollment slightly declined.

In response to the draft review, the region provided enrollment data disaggregated by rural and lowest income quintile. These data indicate modest changes in enrollment between 2007 and 2011.

Indicator2007-08 (%)2011-12 (%)
NER, primary level (age 5-9 years) Rural Girls53.858.4
NER, primary level (age 5-9 years) Rural56.460.7
NER, primary level (age 5-9 years) Lowest income quintile48.051.7
NER, middle level (age 10-12 years) Rural Girls15.019.3
NER, middle level (age 10-12 years), Rural15.722.3
NER, middle level (age 10-12 years), Lowest income quintile11.913.3


Improve the quality and relevance of education: Substantial

Outputs:
The project implemented interventions to reduce the teacher absenteeism rate, since instructional time is an important aspect of student learning. A monthly monitoring system for teachers was established.

Capacity building for school councils was implemented to enhance accountability of teachers to parents, but the ICR provides no evidence to document that this resulted in increased voice for parents or accountability for teachers. Guidelines related to school councils were distributed to all schools. 95% of public primary and middle school councils in 36 districts were provided with capacity development.

54,000 school councils have received annual grants and capacity building support since 2009 (with 97% of councils getting the grants in a timely fashion), but there was a disruption in training due to the floods in 2010, and lack of sustained effort could have decreased positive benefits (ICR, pp. 20- 21). The ICR notes that an impact evaluation of the capacity building program is currently underway.

The project implemented two teacher incentive programs. The "High Achievers Program" rewarded the best-performing schools. This program was discontinued in FY12, as third-party validation showed non-compliance with agreed criteria, and there were fiduciary concerns with the funds transfers. As a result, the "Improvers' Program" was introduced in FY11. In FY12, 1,028 teachers received incentive payments under the Improvers' Program, which rewarded most-improving schools. This program was piloted in 600 schools in three districts (ICR, p. 28). Score cards were distributed in all 600 schools in the 3 pilot districts (ICR, p. 38).

The project supported student diagnostic assessments of learning against grade competencies in Grade 4 and Grade 8 in 2010 and 2011, and examination in Grade 5 and Grade 8. More than 1.4 million Grade 5 and Grade 8 public school children were tested for all three years of the project. In FY11 assessments were also administered in math, social studies, and Urdu in Grade 4, and the analysis was prepared in FY12. The project strengthened the Agency responsible for assessment and improvements were noted in the administration of the examination system; however, there were limited improvements in test design, analysis, and dissemination.

4,754 schools were rehabilitated (water supplies, electricity, boundary walls, and toilets), which exceeded the target (3000 schools). DFID provided financing for rehabilitating an additional 1500 schools. While the disbursement-linked targets were met, there were delays. Third-party validation found concerns with quality of civil works and adherence to the whole school approach (provision of all missing facilities under a single scheme/contract).

1,768 Foundation-Assisted Schools (low-cost private schools) were supported.

Textbooks were delivered to 93% of schools within the first month of the school year, but there are still operational issues and delays at each stage of the process. Monthly monitoring of schools in 2008, 2009, and 2010 showed that between 95% -97% of students had a free textbook (ICR, p. 39). The ICR does not state the number of textbooks provided per grade/subject. There was inadequate use of technical assistance funds to enhance the Punjab Textbook Board to align all textbooks with the new curriculum within stipulated timelines (ICR, p. 29). Six new textbooks were developed. Teacher guides were distributed to all primary schools in the province.

33,299 teachers were recruited based on merit and placed by FY12 in 36 Districts. This entailed the preparation of a recruitment policy, advertisement of postings, display of pre, post, and final merit lists in the Districts, issuance of offer letters, and collection of data (ICR, p. 42). The ICR does not provide evidence demonstrating improved teaching practices or effectiveness.

36 districts provided training and on-site support by the Directorate of Staff Development under the Continuous Professional Development framework, which met the upwardly revised target (36 districts). No information was provided about the effectiveness and impact of the training or number of teachers who received the support.

The teacher accreditation and licensing body was dropped.

Outcomes related to quality:

Teacher absenteeism decreased from 15.1% in 2007 to 10% in 2012, meeting the target (10%). The ICR (p. 20) also indicates that, given the size of the Schools Education Department (over 500,000 employees), mechanisms to establish teacher accountability through the administrative system will remain weak in the medium-term.

An evaluation of the Foundation-Assisted Schools (low-cost private schools) showed that the program increased student achievement (a gain of 0.3-0.5 standard deviations on an unspecified set of indicators) within two years.

The ICR does not report the results from any of the student assessments conducted during the project.


5. Efficiency:

The Project Appraisal Document estimated that the Internal Rate of Return was 9%, based on the expected benefits in terms of lifetime earnings of beneficiaries. The ICR states that this approach did not take into account the impact of institutional strengthening. The ICR calculates the present discount value of expected economic costs, comprising direct and indirect costs (indirect costs: foregone labor earnings of children age 10 or above), to attain school completion. Setting the discount rate equal to 10%, the net Present Discount Value (PDV) is estimated to be PKR 1,964 per child. According to the ICR (p. 18), a positive net PDV is necessary and sufficient for a successful investment operation. However, the calculations in the ICR do not include the repercussions of the floods for school enrollment. The region reported in its reply to IEG's draft review that its calculation of age-earning profiles were based on four population groups (males in urban areas, males in rural areas, females in urban areas, and females in rural areas), which is important given the large gender gap in wages presented in the PAD. It also reported that the calculation were actual changes in the probability distribution of completing different school levels using PSLM 2010/2011.

In the region's reply to the draft review, it reported that there were efficiency gains with the competitive bidding process for textbooks. Price ceilings with the increased number of bidders have resulted in cost savings of US$ 4.5 million per year.

While third party validation found concerns with the quality of civil works, the works largely met standards and were noted to be an improvement over the past.

There was a low utilization of the resources devoted to the technical assistance component (24% of planned), which were needed to build capacity. The underutilization of these resources impacted implementation.

Efficiency is rated Substantial.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
9%
100%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Relevance of objectives is substantial, while relevance of design is modest. Efficacy for two objectives is rated modest, with the remaining objective rated substantial. Efficiency was substantial. The design incorporated a results orientation with disbursement-linked indicators, but there were shortcomings, including faulty design of several indicators and insufficient detail on the technical assistance component. The project implemented a number of activities to improve quality, and teacher absenteeism decreased. The project undertook a number of activities to improve access and encourage girls' participation, but enrollment rates were steady or increased only slightly.


a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:


While progress was made in strengthening education sector institutions in Punjab and a follow-up project is being implemented, there are several risks to the development outcome. First, there are economic challenges facing the province including large budgetary deficits, weak fiscal management, expansionary spending, and a narrow tax base. Second, while the Government is committed to the reform program, its support could erode, given upcoming elections in 2013. Third, capacity remains weak across the provincial departments, sub-departments, and other entities. Governance and fiduciary controls are weak. Fourth, factors such as social tensions and the volatile security situation can reverse progress. Fifth, the Government will need continued support and resources from donors to sustain the reform.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

Project preparation benefited from the lessons of the previous implementation of the reform program and experiences in Punjab, and was based on technical support and experiences from development partners. Lessons learned from Sector Wide Approaches in other countries were incorporated, including: benefits from predictability of the Bank's support through a multi-year results-based investment operation; need to address cross-sectoral issues in planning, budgeting, and expenditure management; and the need for effective monitoring and evaluation to facilitate mid-course program corrections and refinements. DFID provided additional resources to support preparation. To build ownership the preparation process engaged various key stakeholders. There was continuity among the staff who had designed the previous Development Policy Operations and this project. A common M&E Framework was developed, and the indicators and targets were the basis for disbursements by the Bank and DFID. Risks and mitigation measures were properly identified. A Quality Enhancement Review was conducted in February 2009 that endorsed the design, disbursement linked indicators, risk assessment, fiduciary arrangements, and mitigation measures. The ICR (p. 23) points out that this assessment did not fully anticipate weak procurement capacity. There were weaknesses in design and in construction of some of the DLIs (see Section 3b).

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

The Bank team was multi-disciplinary with skills in education, poverty reduction, economic management, financial management, procurement, social and environmental management, communications, and legal issues. There was continuity in the Bank team across the life of the project. The Bank, DFID, and CIDA conducted joint review missions and coordinated to ensure that a single program was supported by the partners to minimize transaction costs for the Government. The partners established an effective collaboration and harmonized efforts. The Bank remained responsive to the government's needs and priorities and provided appropriate technical advice. Despite a challenging security environment, the Bank maintained regular dialogue (and missions) with the development partners, senior level Government officials, and stakeholders at the district level, which supported reform efforts. The Bank provided close supervision and intensive technical support across the reform program and prepared the Additional Financing Agreement when that became necessary. The task team leader (TTL) focused on the most important implementation factors necessary to attain the targets for the disbursement-linked indicators (ICR p. 9). The Bank provided analytical reports and just-in-time policy and technical assistance notes to support reform. The Bank team regularly discussed resolution of procurement issues. Targets that were met during implementation were appropriately upwardly revised. The Bank's Operations Committee reviewed implementation progress after the first year and gave positive comments to the supervision approach and efforts; however, the ICR notes a minor shortcoming: the Bank could have been more proactive in supporting the procurement capacity of the Project Management Implementation Unit to contract firms for technical assistance and ensure the synchronization of technical assistance with the program activities.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

Even after the change in the Punjab Government in 2008, the new Government supported the program. The new Government made education a top priority. The large-scale floods in 2010 understandably diverted attention; however, the Government's efforts to the education reform program remained unwavering despite the political difficulties inherent in some of the reforms (merit-based hiring, transparent recruitment, needs-based placement of teachers, use of public funds to leverage the private sector, student assessments). There was diversion of fiscal resources to relief, reconstruction, and rehabilitation after the 2010 floods, but the Government protected the education budget and core programs, although with occasional delay. Education expenditures for FY10/11 were consistent with the Medium Term Fiscal Framework. The core sector program for FY09 was approved and fully financed. FY10 and FY11 were fully financed after delays in transfers.

There were some shortcomings in performance. The Steering Committee composed of the highest-ranking officials in the province did not meet regularly because of difficulties in organizing meetings, resulting in delayed action on reforms. The Procurement Review Committee that oversaw the contracting of Technical Assistance did not meet regularly, delaying the procurement process and execution of technical assistance.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

The School Education Department and its Program Monitoring and Implementation Unit maintained strong commitment to the program even though institutional changes were required. The ICR (p. 24) reports that the Unit's monitoring of the implementation of the core program was commendable, despite the fact that it was given insufficient staff by the School Education Department for procurement and environmental management. Implementation progress was rated satisfactory or moderately satisfactory throughout the project. All disbursement-linked indicators related to the Reform Program (Component 1) were met, except for the one related to upgrading of schools from primary to middle level, although there were some delays.

There were shortcomings in performance. There was slow implementation of technical assistance activities, as well as underutilization of technical assistance resources, which impacted some aspects of implementation. There was a lack of understanding of the need to synchronize technical assistance (and the procurement of it) with the activities supported by the program. Capacity to implement the upgrading of schools and provision of missing school facilities at the district level was lacking; there was also delay in the release of funds from the province to the district, which delayed implementation on this program. District education monitoring teams were overburdened with other duties, which adversely affected program implementation in activities dependent on data collection. The ICR (p. 9) also reports that implementation was hindered because of a shortfall in education expenditures on the development budget side related to capacity constraints at the district level and diversion of allocated resources. Staff from the Public Works Department did not receive environmental training, thus ownership and understanding of environmental monitoring at the district level was weak.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

Project implementation was to be measured by the key outcome indicators (net primary and middle school enrollment, female/male net enrollment ratio at primary and middle levels in rural areas, primary completion rate, teacher absenteeism rate, and learning levels monitored thorough assessment and examinations) and disbursement-linked indicators. The Project Appraisal Document also included other indicators (pp. 38-39). Some of the indicators were not clear or well defined. There were no measures to demonstrate improved relevance. Indicators to measure equity were inadequate. The Project Appraisal Document contained baseline data and targets related to outcome indicators. The data collection timeline and responsibilities were not specified.

b. M&E Implementation:

Several systems were in place to monitor project implementation and generate data. (1) There was a Monitoring and Information System within the School Education Department and a Project Monitoring and Implementation Unit, which was robust. The data from the Annual School Census fed into the System and was regularly maintained by the Project Monitoring and Implementation Unit. (2) Additional data were collected to monitor school inputs, but the school inspection services were weak (ICR, p. 9). (3) Third party evaluation was used to validate some of the education reforms and indicators (merit based teacher recruitment, missing school facilities, school upgrade program, school council capacity building program, textbook and stipend delivery, public private partnership program, and performance-based incentives for teachers). There was considerable delay in the validation exercises (because of delays in the implementation of Component 2), which reduced their effectiveness. Some of the validation exercises were dropped. (4) There were examinations of Grade 5 and 8 students by the Punjab Examination Commission. The Commission conducted tests of more than one million students in Grade 5 and 8 annually in government schools over the lifetime of the project. The examination process was improved, based on anecdotal reports, but further improvements are still needed in test design and administration, despite the strengthening efforts. Assessment data were augmented with District Teacher Educator visits to examine and track student performance and teaching quality. However, the district education monitoring teams were scattered across a range of activities outside education sector work (population census, relief work, health), which compromised the coverage of schools where data could be collected and monitored. The ICR does not provide results of the student assessments conducted during the project.

Some of the targets were revised in February 2011 with the Additional Financing. Some were reduced due to the negative impact of the floods (primary net enrollment), while others were increased because the original targets were met during implementation (middle-level net enrollment, middle-level gender parity ratio). Household survey data (rather than Annual School Census data) were used to measure the primary school completion rate, since the survey measured completion in all types of schools.

a. M&E Utilization:

The M&E system was reported to have provided reliable data. The Chief Minister's Reform Roadmap relied heavily on the indicators generated by the M&E system and data from the third-party validation efforts to hold districts accountable for performance on some of the indicators. Utilization of the data and M&E system was reported to have increased over the later years of the project. There was analysis of school census and monthly monitoring data, as well as preparation of district-level rankings on key performance indicators. The third-party validation of the High Achievers Program was an important input into the discontinuation of the program in FY2012. School census data collected at the district level was processed, analyzed, and disseminated by the Program Monitoring and Implementation Unit to districts and provincial leadership.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

The project was Category B and triggered OP/BP 4.01, Environmental Assessment. An Environmental Social Management Framework was satisfactorily developed by the Government of Punjab. The Framework addressed potential issues and mitigation measures. The Deputy Director of Planning for the Project Management Implementation Unit and a full time Environmental Officer were charged with monitoring environmental compliance. The ICR reports that these arrangements were adequate and effective, but notes that the pace of implementation improved over the course of the project. Environmental training for 760 district staff was provided in 21 workshops. Environmental safeguard compliance was adequate with supervisory ratings of moderately satisfactory over the course of the project, but the third-party validation found some issues with the quality and timeliness of some of the requirements (ICR, p. 11). Staff from the Public Works Department, which was responsible for the construction, did not receive environmental training, and therefore ownership and understanding of environmental monitoring at the district level was weak.

b. Fiduciary Compliance:

Supervisory ratings for financial management were moderately satisfactory for most of the project's lifetime. There were disbursement-linked indicators related to reductions in advance audit paragraphs in the education sector, which were achieved. Departmental Accounts Committee meetings were held as planned, but were sometimes delayed. Issues in financial management related to staffing the Internal Audit Specialist position (from 2011 and after). Financial reports were not submitted in a timely manner. Despite the technical assistance provided, capacity remained an issue. There was a 73% reduction in outstanding audit paragraphs for FY08 and FY09.

The use of country procurement systems was dropped after delays in the development of necessary regulatory and legislative frameworks in the province. Dialogue was initiated with the Government of Punjab for the improvement of textbook procurement procedures. The Project Management Implementation Unit and School Education Department were at first unfamiliar with the Bank's procurement processes, which contributed to the delay in recruiting consulting firms for the Technical Assistance Component. The Bank team regularly discussed resolution of procurement issues, and these problems were largely resolved by project closing. Simplified guidelines for procurement for School Councils were developed, as well as standard operating procedures for procurement processes and contract management.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Significant
Significant
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

Several lessons are reported in the ICR (p. 24-26), which IEG has synthesized to the following:
    • A results-based design with the use of disbursement-linked indicators can help sector dialogue to focus on the critical factors necessary for achievement of project development outcomes. Such a design can provide development partners with a viable platform to provide consistent policy advice to the Government under a common results framework. In this case, the development partners all engaged in this dialogue. The results-based investment operation improved donor coordination by linking implementation support to government strategy under a common framework and reduced the Government's reporting burden.
    • Strong political commitment and ownership are critical for successfully steering a sector-wide reform program. In this case, there was sustained political commitment, even when the government changed in 2008.
    • Evaluation evidence can reinforce Government commitment. When rigorous evidence demonstrated that the public private partnership initiative in Punjab generated large and quick gains in enrollment and achievement, the Government continued its partnership with the private sector to address shortfalls in education outcomes in underserved rural communities.
    • Strong analytic work is necessary to select indicators, with great rigor towards establishing, measuring, and forecasting the extent of change expected as result of the successful achievement of disbursement-linked indicators. Clear articulation of linkages in formulating indicators and disbursement targets is important to focus efforts on factors that are critical to the achievement of project objectives. The indicators and targets need to be formulated with precision and avoid ambiguity. Ratio-based indicators are sensitive to slight changes in numerator and denominator resulting in volatility and noise, and therefore should not be included.

14. Assessment Recommended?

Yes
Why?
To learn lessons from the use of Disbursement-Linked Indicators, as well as uncover additional data to document attainment of the objectives with the passing of time.

15. Comments on Quality of ICR:


The ICR provides a candid assessment of implementation weaknesses. It describes implementation thoroughly but focuses on indicators rather than attainment of the project's four development objectives. Given the number of evaluations conducted, the ICR could have presented more findings from these reports. The ICR uses a weighting scheme to develop the outcome rating, which is unnecessary as the project development objective was not revised.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions