Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Kh--education For All Fast Track Initiative Catalytic Trust Fund


  
1. Project Data:   
ICR Review Date Posted:
01/14/2014   
Country:
Cambodia
PROJ ID:
P109925
Appraisal
Actual
Project Name:
Kh--education For All Fast Track Initiative Catalytic Trust Fund
Project Costs(US $M)
 57.4  56.51
L/C Number:
Loan/Credit (US $M)
 57.4  56.51
Sector Board:
Education
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  04/10/2008
 
 
Closing Date
06/30/2010 12/31/2012
Sector(s):
Primary education (71%), Pre-primary education (22%), Central government administration (5%), Sub-national government administration (2%)
Theme(s):
Education for all (100% - P)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Jeffery Tanner
Judyth L. Twigg Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:


    This project represents financing from the Education-For-All Fast Track Initiative (FTI) Catalytic Fund for the Cambodia Education Sector Support Scale Up Action Program.

    The stated Program Development Objectives as found in the Grant Agreement and the main body of the Project Appraisal Document (PAD) were succinct:

    To speed up progress towards improved grade 1 right-age entry, net enrollment, retention, progression rates and grade 6 completion rates in primary education.

    A different version of the objectives was indicated in the Results Framework of the PAD. For purposes of the ICR and this ICR Review, the PDOs above will be taken as relevant and binding.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

1: Expanding Early Childhood Education (ECE) (Appraisal: US$4.7 million, Actual: $4.09 million) included three subcomponents:
(i) ECE Facilities Expansion (appraised at US$2.6 million or 4.5 percent of total) aimed to expand and update the stock of ECE classrooms at 650 primary schools and 25 new buildings, as well as expand the annual intake of the Preschool Teacher Training College to 200 teacher trainees.
(ii) Home- and Community-Based ECE Expansion (appraised at US$1.2 million or 2.0 percent of total) was intended to launch 450 new community preschools and 450 home-based preschools.
(iii) ECE Development Policy and Capacity Building (appraised at US$0.9 million or 1.6 percent of total) aimed to update the ECE strategy and conduct community sensitization activities as well as provide regular in-service pedagogical support and supervisory services.

Overall, this component aimed to increase the number of young Cambodians in disadvantaged communities who could benefit from ECE services through the construction of new formal preschool classrooms as well as the development of new community-based and home-based ECE interventions.

2: Improving Primary Education Access and Quality (Appraisal: US$50.7 million, Actual: $47.86 million) included three subcomponents:
(i) Expanding Primary Education Facilities (appraised at US$39.9 million or 69.5 percent of total) to construct 2600 classrooms to complete 650 incomplete primary schools.
(ii) Stimulating Demand for Education (appraised at US$1.9 million or 0.3 percent of total) was to implement a national publicity campaign on right-age entry and scholarships for select grade 3 students.
(iii) Support to Child Friendly Schooling (appraised at US$6.3 million or 10.9 percent of total) aimed to build local capacity to improve school effectiveness, governance, and results by encouraging right-age enrollment, retention, transition to lower secondary education, and improved learning. This was to be accomplished through finalization and roll-out of a school self-assessment tool, in-service training on child-friendly methods, and leadership training for principals.
(iv) School Improvement Grants (appraised at US$1.5 million or 2.6 percent of total) were intended to supplement school operating budgets in areas identified by the self-assessment tool.
(v) Instructional Materials Provision (appraised at US$1.0 million or 1.8 percent of total) was to include the establishment of libraries with age-appropriate library books.

Overall, this component aimed to promote the national goal of universal primary completion through the construction of new primary classrooms and provision of demand-side incentives, while promoting child-friendly schooling initiatives to enhance educational quality and strengthen teacher professional capacity.

3: Promoting Institutional Development and Capacity Building (Appraisal: US$2.0 million, Actual: 4.56 million) consisted of two subcomponents:
(i) District Offices of Education Facilities Expansion (appraised at US$0.9 million or 1.6 percent of total) was to build 60 new District Offices of Education buildings, inclusive of furniture, in rural areas where there had been none.
(ii) Financial Management, Procurement and Program Management Capacity Building (appraised at US$1.1
million or 1.9 percent of total) was to strengthen internal and external audit, accounting, reporting, transparency, and decentralization of procurement. Capacity building was to occur with select province-level staff on accounting, record keeping, and financial monitoring. A client construction unit was also to be established to monitor the public works construction of the project.

Overall, this component was intended to enable decentralized education management and monitoring by Departmental and Provincial Offices of Education and the Inspectorate of Education through the rollout of a new school inspection policy, geared towards enhancing direct support services and responsiveness to school needs, and through the strengthening of administrative processes for procurement and financial management at all administrative levels.

An additional sub-component of an Early Grade Reading Assessment was added to Component 2 during the project’s third restructuring in December 2010. These assessments were to be carried out in grades 1 and 2 and were requested by the Ministry of Education Youth and Sports (MoEYS) following a World bank workshop on early grade reading.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Cost:

The original total commitment was US$57.4 million, and the project stayed at that amount despite being restructured three times. Differences in disbursement amounts reported in the ICR of $57.36 million (p. i) and $56.51 million (p. 26) are due to the former not accounting for the 4-month grace period in the Bank disbursement letter.

Significant deviations in expenditure -- lower than planned for Components 1 and 2, and higher than planned for Component 3 -- are attributable to several issues, explained by the project team. First, construction costs rose by 50-60% over the project period (ICR, p. 7). Second, because of those higher costs, additional classrooms in Component 2 were done as rehabilitations or add-ons to existing schools rather than new free -standing schools. No such reduction in costs by adding onto existing structures was possible for the construction of the district offices specified in Component 3. Third, a salary supplement for a wage increase at a rate harmonized across all donors, termed Priority Operating Costs, was added in the second restructuring, and its costs accrued largely to the Client Construction unit in Component 3. Finally, nearly twice as many trainings and workshops as planned at appraisal were accomplished, most of which accrued to Component 3.

Financing:

The Fast Track Initiative Catalytic Fund provided US$57.4 million in grants. Actual disbursement was US$57.36 million. The ICR is unclear regarding the remaining US$40,000. No amount was cancelled during implementation. With the revised start dates and delays, the initial disbursement amount of US$18 million planned for 2008 Q4 was not realized until 30 months later in 2011 Q2; from that point, however, disbursement followed the slope of the anticipated trajectory fairly closely.

Borrower Contribution:

There was no planned Borrower contribution.

Dates:

    • The project was approved on 10 April 2008, and the effectiveness period began 13 June 2008.
    • The project was restructured three times, resulting in the extension of the project closing date from 30 June 2010 by a total of 30 months to 31 December 2012.
    • 27 April 2010: A 24-month extension to the original project closing date, from 30 June 2010 to 30 June 2012, took place because the General Election prevented the Ministry from appointing key project personnel in a timely fashion and because of delays in completing construction activities.
    • 19 December 2010: A restructuring replaced planned salary supplement and incentive schemes with the Priority Operating Costs scheme and instituted the Early Grade Reading Assessment sub-component.
    • 28 June 2012: A 6-month delay in project closing, from 30 June 2012 to 31 December 2012, was approved to recover lost time due to flooding and to repair damage to project infrastructure caused by the flooding.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
High. The most recent Cambodia Country Assistance Strategy (2005-2008) includes two pillars: removing government constraints to attaining the Cambodia Millennium Development Goals (CMDGs), and supporting the formulation and implementation of a government-led strategy and investment program for achieving the CMDGs. While the Project Development Objectives did not explicitly address the first pillar, the way the project was implemented -- through the Government’s institutions, most notably the Ministry of Education, Youth, and Sports -- is certainly relevant in improving government capacity, which can in turn reduce governance constraints.

The second pillar focuses on two objectives: supporting a government-led strategy to achieve the CMDGs, and supporting sectors and sub-sectors critical to achieving the CMDGs. The project is more directly aligned in its objectives and design to this second pillar, with respect to the second Cambodian Millennium Development Goal to achieve universal nine-year basic education. This CMDG has 10 targets covering admission, enrollment, survival (retention/progression), and literacy rates as well as the proportion of children out of school and gender ratio disparities. Of the five main objectives listed for the project, four are directly related to these ten CMDG indicators.

At the time of appraisal, the education system in Cambodia had made great strides in attendance (91.3% in 2006) but was still deficient in several important areas. While the correct age for entry into primary school was six years, Cambodian children entered quite late, at age 7.8 years on average. A gender disparity in secondary graduates was attributed to differential progression rates from early in the education cycle. Moreover, primary completion rates of 85% were even lower for poor children. Repetition rates had seen significant progress in the decade prior to the project, dropping from 26.3 to 11 percent. The pupil:teacher ratio was stagnant and high at around 50, and there was real concern over low teacher capacity. Learning outcomes were poor as measured by student knowledge of the official Khmer and mathematics curriculum. Finally, while the government strategy called for a move toward decentralization, local capacity for leading and improving education was weak. The project was therefore highly relevant to country conditions at the time of appraisal.

b. Relevance of Design:

Modest. The project’s design (its planned activities and policy areas) is consistent with its stated objectives and is well-informed by solid research on both the local Cambodian context and global best practice. Still, the Results Framework of the PAD is under-developed and inaccurate. The design also did not sufficiently take into account capacity constraints of the Ministry of Education, Youth, and Sports (MoEYS).

The project’s three components of early childhood education, improving primary education access and quality, and promoting institutional development and capacity building reflect well the PDOs of the project. The first two subcomponents directly link to the five outcomes of right age of entry at grade 1, net enrollment, retention, progression, and completion. The third subcomponent of promoting institutional development and capacity building within the Ministry of Education is less directly related to the outcomes, but is not orthogonal.

Clearly the design of the first two components in particular benefitted from a careful and thorough understanding of the best available evidence on early childhood education and primary school attendance. It relied heavily on both solid impact evaluation evidence from other contexts and data and analysis specific to Cambodia.

However, while the design addressed demand-side constraints to primary education with its scholarship program, it did not consider the demand constraints (in addition to the planned-for supply constraints) that inhibit families from taking advantage of early childhood education (ECE) programs. Moreover, the design did not consider that the (apparently not uncommon) current practice of children beginning to attend primary school a year early at age 5 would have positive benefits for the child over and above what an ECE curriculum could provide, thus lowering children’s cognitive development by taking them out of primary school and putting them in ECE interventions. Similarly, the design (and the ICR) also did not consider the effects and interactions of other Cambodian programs contemporary to the Education Sector Support Scale Up Action Program (ESSSUAP).

Finally, the design did not account for exogenous factors that affected the implementation of the project. As discussed in the ICR, the design gives significant ownership to, and so places a heavy burden on, the Government (specifically the MoEYS) and contractors to detail and carry out the planned activities. The 24-month delay in implementation was due largely to issues around a General Election that could have been taken into account in the design of the project.


4. Achievement of Objectives (Efficacy) :


Objective 1: To speed up progress towards improved grade 1 right-age entry. Modest.

The project did not include an indicator for Grade 1 right-age entry in its monitoring plan. However, several program indicators hint that the level of grade 1 right-age entry may have improved under the implicit program logic that primary school age-at-entry can be changed through early childhood education. ECE enrollment of 5 year olds and 3-5 year olds both increased and essentially met their targets. Primary school net admission rates also increased from a high base. Whether these improvements were increases over the previous trend (e.g. “speed up progress”) is also not known, as only 1 baseline period is given in the ICR.

The ICR also does not discuss other factors that may have led to the outcomes reported below; indeed, the ICR points out that while these are national-level trends, students within the project area accounted for no more than 6% of the total student population.

Outputs
    • The project built 424 early childhood education classrooms of the targeted 675, achieving 62.8% of the original target value.
    • The project launched all of the targeted 450 community preschools.
    • The project also supported 450 new home-based education programs, meeting the target of 450.

Outcomes
Early childhood education enrollment of 5-year olds increased nationally from 31.4% at the end of CY 2008 to 52.66% at the end of CY 2012, exceeding the 50% target for the program.
Early childhood education enrollment of 3-5-year olds increased nationally from 15.57% at the end of CY 2008 to 28.89% at the end of CY 2012, below the 30% target for the program.
Net admission rates (age group not given, though grade 1 is indicated) increased nationally from 89.5% at the end of CY 2008 to 94.1% at the end of CY 2012, not reaching the target of 100%. The high base of admissions (nearly 90% at baseline) suggests that getting children into school is not as much of a challenge as students entering at the right age. No age data are given for those entering grade 1.
Though the monitoring system did not collect data on enrollment age, an impact evaluation (Bouguen et al 2013) on the project presents convincing evidence that the program was indeed responsible for increasing the number of children who entered primary school at the right age. Unfortunately, it did so at the cost of lower cognitive attainment among recipient children. A large portion of the children in the relevant counterfactual group attended primary school a year early—at age five—and so received greater cognitive stimulus at primary school than those in the Bank-funded ECE program.

Objective 2: To speed up progress towards improved net enrollment. Substantial.

Though the project did not reach the target of 100% national enrollment, it was unlikely to do so working with 6% of the student population. Still, overall enrollment did increase, and two well-constructed impact evaluations demonstrate that the scholarship program improved attendance, a distinct though related construct to enrollment. Students receiving program scholarships were more likely to be in school and attended more hours of school.

Outputs
The project awarded 21,780 scholarships, or 91.12% of the targeted 24,000 scholarships. These scholarships were awarded conditional on enrollment.

Outcomes
The net primary enrollment rate increased nationally from 93.3% at the end of CY 2008 to 96.4% at the end of CY 2012, 3.6 percentage points below the 100% target for the program and covering 46.27% of the gap to the target.
An impact evaluation of the scholarship program by Barrera-Osorio and Filmer (2013) asserts that both the merit-based and the means-based scholarships increased “enrollment and attendance” over equally qualified candidates randomly assigned to a control group. The impact evaluation uses three proxies for enrollment and attendance, including “the proportion of students reaching 6th grade” and “the highest grade completed” (p. 14). The paper notes that “the intervention is aimed at directly incentivizing higher enrollment and attendance rates, so scholarship students must stay enrolled… until they graduate from primary school (6th grade)” (p. 14). Recalling that the problem of net enrollment in Cambodia is far more concentrated in the later grades than in the earlier grades (only 60% of students in the control group made it to sixth grade or beyond, compared to a 93% net primary enrollment rate nationally), the scholarship program, which began in the middle grades and resulted in a large and highly significant increase in the proportion of students reaching grade six of 18 and 13 percentage points for the poverty and merit-based programs, respectively, is highly plausible to have substantially sped up progress towards improved net enrollment among those in the project’s area. The scholarship program is also similar in design and effect to a Cambodian Lower Secondary School scholarship program that increased enrollment on the order of 20-25 percentage points (Filmer and Schady, 2008, 2011).

Objective 3: To speed up progress towards improved retention. Substantial.

Retention is interpreted in the PAD as “retention of students in upper primary” (p. 23). Though the program did not collect indicators or monitoring and evaluation data on retention, evidence from a well-executed impact evaluation gives strong support that the program caused better attendance among 6th graders, a likely close correlate to retention.

Outputs
In addition to the Output of Objective 2, the project completed 424 of formerly “incomplete” primary schools, reaching 2/3 of the target of 650 schools. The ICR partially attributes this shortfall to a change in tactics to also include construction (expansion) of overly crowded schools. Increasing the number of schools with upper primary schools by completing incomplete schools is presumed to have an effect on improving retention / enrollment / attendance of upper primary grade students.

Outcomes
The Barrera-Osorio and Filmer (2013) impact evaluation estimates that the poverty-targeted scholarships of the program increased the probability of reaching 6th grade by 18 percentage points.
The Barrera-Osorio and Filmer (2013) impact evaluation estimates that the poverty-targeted scholarships of the program caused students to complete 0.33 more grades.

Because retention is a necessary condition for increasing the number of grades completed by a population, by substantially increasing grades completed (and the likelihood of reaching 6th grade), the scholarship component of the program very plausibly improved retention of students in upper primary.

Objective 4: To speed up progress towards improved progression rates. Substantial.

“Progression rates” is not further defined in the policy documents, but may reasonably be interpreted as moving along in grades completed and avoiding repetition. The effort to increase the number of complete primary schools is implied to improve progression by providing a class for students to progress into. Also, the inferred (though unstated) program logic of the scholarships, especially merit-based scholarships, is that rewarding students for attending may help them progress in their education.

Outputs
The relevant outputs are the same as those in Objective 3.

Outcomes
The program had a target indicator of reducing grade 1 repetition from 18.7% to 4%. Grade 1 repetition rates actually decreased to 10.9% nationally. It is unclear how much of this decline is attributable to the program, given the low share of students nationally who were involved in the program.
The Barrera-Osorio and Filmer (2013) impact evaluation estimates that the merit-targeted scholarships of the program increased the probability of reaching 6th grade by 13 percentage points.
The Barrera-Osorio and Filmer (2013) impact evaluation estimates that the merit-targeted scholarships of the program caused students to complete 0.2 more grades than those in the control group.

Objective 5: To speed up progress towards improved grade 6 completion rates. Modest.

The program logic of grade 6 completion, as distinct from the other objectives, is not clear. The project design seems to imply that building more classroom capacity and an increase in the number of “complete” primary schools. Similarly, providing scholarships may have induced students to complete the sixth grade. The indicators for this outcome are taken from national-level data, and the influence of the Bank project, given its low number of enrolled students, is not clear.

Outputs
The relevant, directly related outputs are the same as those listed in Objectives 3 and 4.

Outcomes
    • The project’s indicator for this objective, primary completion rate, increased from 86.25% at the end of CY 2008 to 89.7% at the end of CY 2012, well short of the target of 95%. The increase in primary completion rates of less than 3.5 percentage points covered only one third of the intended distance from baseline to target.
    • The Barrera-Osorio and Filmer (2013) Impact evaluations established that the scholarship programs increased the number of grades completed and the probability of reaching grade 6, which are necessary but not sufficient conditions for increasing grade 6 completion rates. Because the scholarships increased the probability of reaching grade 6 and the total number of years completed, it may be that the program increased grade 6 completion rates. However, this is not clearly established.

5. Efficiency:

Modest. The project was not subjected to an analysis of economic efficiency at appraisal. However, the ICR conducts a cost-benefit analysis of Components 1.1 and 1.2 on Early Childhood Education (facility and home- and community-based ECE expansion) and Component 2.2 on stimulating demand for education (the scholarship program). Together these components account for US$5.7 million, or 10% of the total value of the project. The internal rates of return for all of the evaluated components are above the presumed opportunity cost of capital (10%). This indicates substantial levels of efficiency, and reasonable confidence around those estimates, for these components.

Point Value (Range) Coverage / Scope
Component 1.1-1.2
ECE Facilities, Home, Cmty Expansion
15.48% (14.77 - 18.31%)
6.6%
Component 2
Stimulating Ed Demand (Scholarships)
10.89% (8.1 – 13.88%)
3.3%


The quality of the ICR's economic analysis itself is quite high. It does commendable work in clearly laying out the assumptions of the analysis and attempting to make a full accounting for both direct and indirect costs, using estimates of the benefits based on sound empirical studies, making reasonable projections, and conducting sensitivity analysis.

The large enrollment and attendance effects of the CCT scholarship program were attained at an extremely small cost (US $20 per year) when compared with the more modest gains and significantly higher costs of similar programs worldwide (Fiszbein and Schady, 2011; Filmer and Schady 2010).

Still, the ICR seems to indicate that the 25-year time horizon does not include the two-year implementation delay as it should. And while it cites an empirical study on the UNICEF-funded ECE program in Cambodia that reports cognitive improvements in 3-5 year olds, an impact evaluation specifically on the Bank-funded ECE programs indicates that while younger children made modest gains, 5-year-olds did worse.

The internal rates of return for all of the evaluated components are above the presumed opportunity cost of capital (10%). This indicates substantial levels of efficiency, and reasonable confidence around those estimates, for these components. The analyzed investments seem to have been cost effective and well spent, as the ECE and CCT programs represent the class of interventions with the best scholarly work demonstrating the highest-known impacts. However, because the evaluated components constitute only 10% of the total project cost, and in considering the challenges noted above, the efficiency of the entire project is rated Modest..

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
Yes
13.95%
10%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


The overall outcome is rated Moderately Satisfactory.

The relevance of the objectives is high and overall, the operation’s relevant objectives were achieved. For three of the project's five objectives, the ICR presents high-quality direct evidence that the Bank’s programs caused significant improvements in Cambodian education, leading to a substantial efficacy rating. However, achievement of right-age entry to grade 1 and completion of grade 6 are rated modest due to an inadequate results framework leading to insufficient monitoring and indicators for these two objectives.

The modest relevance of design reflects the non-negligible disconnect between the official PDOs and the results framework, though the ICR claims that disconnect was initially an artifact of the global EFA-FTI guidelines’ prohibition of project-specific objectives separate from the Government’s overall goals (p. 9, footnote 3). Efficiency is rated modest because of the large share of the funding that was not cost-benefit analyzed, though the elements that are evaluated have a solid reported ERR from a robust model. Moreover, comparing elements of the Cambodian operation to similar programs elsewhere in the world underscores the substantial level of efficiency of those evaluated elements.

Taken together, these ratings are indicative of moderate shortcomings in the project's preparation and implementation, leading to an Outcome rating of Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:


A number of critical activities, such as the school management program and support for early grade reading practices and assessment, are being continued under the ESSSUAP. The ICR indicates that project management responsibilities were integrated within the MoEYS, including the scholarship program, the teacher and director training, the National Learning Assessment, the training module on inclusive education, and the Early Grade Reading Assessment (EGRA) methodologies and activities.

These subcomponents are all institutional in nature and housed in the MoEYS. There are also political risks to the development outcome, as the start of the program was delayed for 24 months due to knock-on effects from a general election. There seems to be some risk that the MoEYS will be able to continue these programs in the event of future institutional changes, electoral or otherwise. The MoEYS was given an overall Risk Assessment at baseline of High; while the project certainly aimed to improve the capacity of MoEYS and did a significant amount of training, there is no information on whether or not that training has actually changed outcomes or behaviors, or subsequently, risk to development.

The ICR does not provide information on the maintenance of the hard infrastructure built or provided over the course of the project. Furthermore, the project sought an extension of six months because flooding had washed out elements of project infrastructure. It is not clear what contingency plans or funds exist should a similar event transpire in the future.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

While the PAD had a robust discussion of efficacy, its assessment of efficiency was quite limited. Moreover, the Results Framework was not fully developed. Similarly, the planned system of monitoring was inadequate, relying heavily on national level statistics for the main indicators rather than making provision to monitor outcomes among the project’s target groups; while the planned impact evaluations are a tremendous resource ex post, they do not in themselves constitute a monitoring and evaluation system. As acknowledged by the ICR (p. 19), the disconnect between the Grant Agreement / PDO objectives and the objectives in the Results Framework (with its program logic that can only be inferred) led to a lack of measurement of key elements of the objectives. Delays in implementation were caused by a mismatch between required and available capacity, as highlighted by the after-effects of the election; while this capacity disjunction may be interpreted as insufficient understanding of the Cambodian political economy on the part of the Bank, designing (and delaying) the program so that it was run by the Cambodian government was born out of the entirely appropriate objective of improving MoEYS capacity and ownership of the project.

Notwithstanding these challenges, the project's design was clearly well founded and based on the available knowledge and indicators outlined in the country's education strategic plan (which had been endorsed by donor partners). The preparation team integrated significant knowledge of the Cambodian context with rigorous evidence in developing a project designed to address major elements of the Cambodian Millennium Development Goals. Though there were significant delays in the beginning, the fact that the project was eventually launched is likely partially attributable to the Bank's close relationship with the government in the design phase and the resulting high degree of relevance of the overall strategy to the government’s stated goals. The project also set out to produce new diagnostic tools and rigorous evaluation evidence that contribute to the global public good of knowledge.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

The length of term of the Task Team Leader provided both access to the Ministry and good knowledge of the intricacies of the Cambodian context. The continuity provided by having a single Task Team Leader throughout the project likely contributed to the ability of the project to be successfully launched despite a 24-month delay in the effective start date. The project benefitted from 9 supervision missions, averaging two annually, with the appropriate staff as needed: procurement, financial management, safeguards, social assessment, and engineering experts. The team is reported to have been proactive, raising appropriate issues over the lifetime of the project. The Grantee's comments on the Draft ICR concur that the Government enjoyed “close technical assistance and support from the World Bank task team,” calling it “crucial in developing the EGRA programs and in designing the highly complex prevalence study on out-of-school children” (p. 46).

Yet delays occurred from the outset, and the risk assessment at entry (referred to but not described in the PAD) indicates an overall Risk Assessment of the MoEYS of High. Accordingly, though the project appropriately aimed to increase the capacity of the MoYES, the Bank could have anticipated a rough start and assumed a greater role in helping to establish the structure and function of the project management team prior to and early in implementation. Enhanced supervision would seem to have been warranted during this time.

More importantly, in spite of restructuring the project three times, the Bank never formally revised the objectives and the results framework indicators to bring them into harmony. As a result, two of the five project objectives were not mentioned in the results framework and did not have relevant indicators collected about them. Moreover, a mid-term review, though indicated in the PAD, was never conducted. Several supervisory opportunities to strengthen the relevance of the design to the objectives were therefore missed. The ICR also indicates that greater candor was needed in the ISRs beyond indicators of disbursements and construction progress to assess progress against project-level (rather than national-level) indicators.

Taken together, these are significant shortcomings in the proactive identification of opportunities and resolution of threats, but in light of the positive response from the Grantee, seem not to have escalated to the level of major shortcomings.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The government was overall in compliance with project covenants and agreements. It built a solid strategy based on previous policy experience and established sound sectoral goals. Continuity in having only one Minister of MoEYS through the project's life cycle and one Project Director at ESSSUAP during the implementation cycle played a stabilizing role and created an enabling environment. The fact that the project was fully prepared by MoEYS staff indicates strong government ownership and commitment to achieving the development objectives.

The delays in implementation due to an election may also be partially attributable to the Government. The inability to formulate a project management team from the outset contributed to a two-year delay.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

MoEYS successfully integrated learning from previous experience into the design and execution of the project. The ICR indicates that once operations began, the MoEYS was a full and willing partner, highly engaged in various aspects of the project and Bank missions throughout the life of the project.

Perhaps because the Early Childhood Education Department, the Internal Audit Department, and the Quality Assurance Department were all new units, there were challenges and delays with ECE implementation. This inadequate capacity at entry is also likely responsible for inadequate contract management, which led to delays in construction activities because of non-performing contractors. The ICR also points out that weak capacity and inconsistent payments of the merit-based performance incentives slowed implementation activities.

The ICR indicates that the MoEYS was “adequately staffed” (p. 22) yet also points to delays in staffing as a result of an election as the primary reason that operations were delayed in starting for two years.

Finally, MoEYS did not regularly report outcome data specifically on students affected by the project. This became particularly troublesome for the ECE component in which an impact evaluation (Bouguen 2013) concluded that the component’s lack of effect was largely due to implementation issues, including delays in building facilities, limited incentives for teachers and volunteers, and low and irregular stipends for teachers. These issues could have been highlighted by a better monitoring system and potentially resolved.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The M&E design suffered from a disconnect between the PDOs used in the Grant Agreement and main body of the PAD and those used in the Results Framework in Annex 2 of the PAD. The Results Framework misstates the PDOs as “assist[ing] MoEYS to (a) expand preschool and primary education services so as to reach the most vulnerable groups of children through both demand- and supply – side interventions and (b) enhance professional and administrative institutional capacity for education service delivery at central, provincial, district and school levels” (PAD, p. 48). Moreover, while the Results Framework does sketch out intermediate results and indicators, it gives little detail and does not give clear links running from objectives through activities to intermediate and final outcomes.

The PDOs in the PAD Annex were used to design the M&E system. The ICR points out two results of this shortcoming: a failure to measure the right outcome indicators, and a failure to measure the right outcome population. The indicators did not measure outcomes on age at Grade 1 entry or retention (drop-out rate or share of children in/out of school). They also did not measure the population affected by the project, opting instead for national-level data even as the Bank’s project directly affected some 6% of the school-age population. The M&E framework remained inadequate as outcomes were not tracked for the project's target groups, a sub-directive of the Grant/PAD PDOs.

There is evidence that the design of the monitoring system for the ECE program was inadequate in its design to aid in making timely adjustments along the way in implementation.

While the design of the M&E system was inadequate for the project overall, the task team did design
several high quality impact evaluations, including randomized control trials, as embedded evaluation elements within a handful of the specific project activities to be able to assess the effectiveness of key parts of the intervention, a rare and extremely welcome step. Yet however well designed these impact evaluations may be, they cannot replace a well-designed project-wide monitoring system.

b. M&E Implementation:

The M&E appears to have been implemented as planned. That included updates to key indicators through the MoEYS’s Education Management Information System, updated annually. It seems unlikely that this level of frequency would be sufficient to inform the project of needed course corrections. The ECE components, for example, were cited as lacking sufficiently detailed and frequent monitoring information with which to aid in making timely adjustments during implementation. Still, in spite of the lack of data in the M&E system on outcomes, the system did deliver semi-annual reports on a robust set of outputs.

The ICR (p. 9) cites 6 classes of evaluations carried out during the life of the project. Yet it cites only two of those in the reference list to the ICR, and none in the main body of the ICR itself, making it difficult to assess how well these evaluations were implemented. The impact evaluations located are of quite high quality and provide clear information on the attributable effects (or lack thereof) of the intervention.

a. M&E Utilization:

The ICR cites several examples of how the various impact evaluations done on the project have impacted Cambodia’s education plans moving forward. However, it provides no evidence of how monitoring and evaluation data were used during the course of the project to make improvements during implementation. Indeed, one impact evaluation suggests that the serious deviations from design during the implementation of the ECE components could have been observed and potentially corrected with a robust M&E system. Without that system, the ECE component yielded disappointing results of no significant improvements for children in the program.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
There were no safeguard policies triggered by the project. There were no major adverse environmental impacts.

b. Fiduciary Compliance:

Procurement: The ICR (p. 12) reports that procurement specialists monitored the project regularly and found procurement to be consistently satisfactory, though there were initial delays at startup partly due to lack of familiarity with procurement procedures among the MoEYS team. MoEYS staff received training through the initial stages from the Bank, but by the end the MoEYS became the first Ministry to move away from the arrangement of the International Procurement Agency handling all fiduciary matters, to allow MoEYS to handle procurement directly in subsequent IDA projects.

Financial management: The ICR (p. 11) reports that financial management staff monitored the project regularly, including attending supervisory missions, and found financial management to be consistently satisfactory or moderately satisfactory. Audits were submitted regularly and were unqualified.

c. Unintended Impacts (positive or negative):

None reported in the ICR, though the ICR could have explored this issue further (e.g. budget fungibility, effect of delays on children who aged out of beneficiary age, demonstration/spillover effects). One impact evaluation of the project indicates that the Bank’s program caused average cognitive acuity of five-year-old children to be lower than what it would have been if the program had not existed.

There were several outputs of the project that were not explicitly tied to the original PDOs, though they do appear in the results framework. From the perspective of the original PDOs, these may be considered appropriately as “unintended impacts”:

    • Financing and provision of materials on Early Grade Reading Assessments
    • Strong consideration of vulnerable and disadvantaged groups, including Khmer and girls, given throughout all phases of the project
    • Scholarships targeted to the disadvantaged and to girls
    • Support of a study to determine prevalence of visual and auditory impairment among students
    • Development of new monitoring and evaluation for ECE supervision
    • Issuance of school self-assessment and planning tools for Child-Friendly Schools
    • Improved capacity of education infrastructure and human resources
    • Granting 606 of targeted 650 School Improvement Grants
    • Providing 590 of targeted 650 schools with age-appropriate library books
    • Construction of all 60 targeted district offices of education
    • In-service pedagogical support for 450 Core Mothers, 450 Community Teachers, 318 formal preschool teachers, and 40 MoEYS staff
    • Training of 46,695 teachers in Child-Friendly Schools methodologies, more than tripling the target of 15,000
    • Training of 505 teachers and 1,307 staff in District or Provincial Offices of Education on the use of a Child-Friendly Schools checklist to identify strengths and areas for improvement
    • Provision of leadership training to 2,861 principals, nearly double the targeted 1,500
    • Provision of Financial Management and Procurement training to 4,690 MoEYS administrators, nearly 16 times more than the targeted 300.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Significant
Concerns remain over institutional and structural risks. There is limited evidence on how the MoEYS may have improved from a baseline risk assessment of High. The ICR does not address contingencies to continuing environmental risks to infrastructure. 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The ICR offers 6 lessons (pp. 22-23). The three most important are highlighted below, with additional observations by IEG:

“An appropriate and thorough institutional capacity and readiness assessment during the preparation stage is strongly recommended to improve the implementation readiness conditions for complex initiatives being piloted for the first time.” IEG concurs with the ICR assessment that “implementation issues may have been mitigated by a more thorough capacity assessment” and “a capacity assessment of the construction department could have helped set more realistic civil works targets” (p. 23). Such assessments would be an important first but insufficient step in avoiding the starting delays and other implementation challenges encountered by the project. Efforts to proactively plan ways to mitigate weaknesses revealed by those assessments are also required.
“A well-conceived ‘results-chain’ demonstrates connectivity between the stated PDOs and the agreed ‘observable and measurable’ outcome indicators, and the attribution is limited to the scope of the project.” The project lacked such a results chain, and what monitoring system it did have deviated from the PDOs. Management should have taken the opportunity of one of the three project restructurings to harmonize the discord between the PDOs and the Results Framework. The ICR also makes the useful point that “candid and internally consistent ISRs should serve as a critical tool for management to encourage proactivity” (p. 23).

“Targeting students with high academic potential—while ensuring that the poorest students are among that set—may be an approach that maximizes both the equity and effectiveness objectives of demand-side incentive programs.” Where the first two lessons treat issues of project management and implementation, this lesson is a critical design finding from a well-executed rigorous impact evaluation that should inform the design of future demand-side incentive programs both in Cambodia and elsewhere. The ICR also calls for further research into whether this lesson “hold[s] in other settings (e.g. where there is greater heterogeneity in student poverty levels)” (p. 23).

14. Assessment Recommended?

Yes
Why?


The project was the subject of five impact evaluations. It would be useful to follow up to see how the results from these evaluations are being used in practice to inform subsequent policy design. These impact evaluations also present an opportunity to field a follow-up round of surveys for an impact evaluation aimed at understanding results over time. A follow-up assessment could also explore the impact of the project's school construction and training/capacity building. Finally, this project could be usefully compared with other FTI or education SWAp projects.


15. Comments on Quality of ICR:


The ICR is generally candid about shortcomings in the project's monitoring and evaluation framework, and it provides some data beyond the project's planned indicators. However, the overall evidence base from the project itself (outside of the impact evaluations) remains only marginally sufficient to allow proper assessment of the project's outcomes (outcomes for nationwide population ascribed to the small proportion of the population directly affected by components 1 and 2). The seven previous evaluations done on the project are mentioned in the ICR (p. 9), but their findings are not cited and most are not listed in the ICR bibliography. Importantly, one evaluation that is not cited in the ICR, Bougan et al 2013, indicates that the ECE components of the project did not yield positive results.

The ICR's section on outcomes could have been better arranged by objective, and the results of the impact evaluations could have been better and more explicitly employed. There is little discussion in the ICR of a counterfactual or other factors that may have influenced national trend data.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions