Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Fast Track Initiative Catalytic Grant Fund -2


  
1. Project Data:   
ICR Review Date Posted:
01/14/2014   
Country:
Kyrgyz Republic
PROJ ID:
P118423
Appraisal
Actual
Project Name:
Fast Track Initiative Catalytic Grant Fund -2
Project Costs(US $M)
 6.00  5.90
L/C Number:
Loan/Credit (US $M)
 6.00  5.90
Sector Board:
Education
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  03/24/2011
 
 
Closing Date
08/31/2012 12/31/2012
Sector(s):
Pre-primary education (91%), Public administration- Education (9%)
Theme(s):
Education for all (100%)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Peter D. Bachrach
Judyth L. Twigg Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The Grant Agreement (p. 4) stated that the objective of the Project was to support the Recipient’s introduction of the national school preparation program for a smooth transition to school for children between the ages of five (5) and six (6) years. The PAD (p. 11) added other elements, stating that the project’s objective was to support the government’s introduction of the mandatory national school preparation program for 5-6 year old children by improving learning and teaching conditions in the program and its institutionalization within the education system.
    Because the Grant Agreement statement of the objective (“smooth transition”) is vague, this Review uses the more operational phrasing in the PAD (“by improving learning and teaching conditions in the program and its institutionalization within the education system”) to present the results chain.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

The project comprised three components:

Component 1: Quality enhancement of the mandatory preparation program (Appraisal: US$2.26 million; Actual: US$2.16 million or 96 percent of appraisal) was to include: (i) teacher training in preschool pedagogies for 3,200 primary school teachers nationwide, to prepare them for teaching in a mandatory school preparation program for 5- and 6-year-olds; and (ii) training of regional education department methodologists in mentoring, and providing mentoring to trained teachers to facilitate classroom application.

Component 2: Improving the learning environment (Appraisal: US$4.7 million; Actual: US$4.7 million or 100 percent of appraisal) was to include: (i) a standard package of learning and teaching materials, including developmental games, reading books, visual aids, and methodological materials sufficient to support the school preparation course; and (ii) children’s and teachers’ tables and chairs, blackboards, and storage shelves or cupboards for the provided resources.

Component 3: Policy, research, evaluation, and project management (Appraisal: US$0.64 million; Actual: US$0.64 million or 100 percent of appraisal) was to include support for measures: (i) to facilitate policy development for the mandatory school preparatory course; (ii) to institutionalize the new practices within education structure and management functions; and (iii) to carry out public awareness and advocacy initiatives to inform and orient parents to the importance of sending their children to the preparatory program. The component also was to support measures to ensure that Grant activities were implemented on time and in a satisfactory manner.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project cost: The project costs were virtually identical with the appraised costs.

Financing: The financing instrument for the Fast Track Initiative Catalytic Fund Grant 2 (FTI 2) was a specific sector investment Grant in the amount of US$ 6.0 million. FTI 2 was a follow on project to the Fast Track Initiative Catalytic Fund Grant (FTI 1), which was approved in May 2007 (in the amount of US$9 million) and closed in May 2010.

Borrower contribution: The planned counterpart contribution of US$ 1.6 million was paid in full.

Dates: The project closing date was extended by four months, from August 31, 2012 to December 31, 2012, to enable the Government to use savings accumulated during implementation to purchase additional learning materials.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Relevance of Objectives: High

The project's objectives were highly relevant to country conditions at the time of appraisal. National assessments of learning (2007 and 2009) and international data from the Program for International Student Assessment (PISA, 2006) revealed low levels of achievement among primary and secondary students. Further, a comparative analysis of PISA results in Central and Eastern Europe as well as the Commonwealth of Independent States indicated that countries performing poorly on PISA shared many similarities, including poor preschool coverage.

The Project’s objectives were also highly consistent with: (i) the Law on Preschool Education (2009); and the Government’s Strategic Program of Education System Development (2008-2011). Both prioritized access to affordable, high quality preschool preparation programs and equal opportunities for each child to prepare for schooling. The objectives were also consistent with: (i) the Joint Country Support Strategy (JCSS) of the principal development partners, including the World Bank, for the period 2007-2010; and (ii) the implementation of the Country Development Strategy (CDS) covering the same period and highlighting human development as a key national objective, with a particular emphasis on access to increased and affordable education services.

Currently, the project’s objectives support both the Government’s Education Strategy (EDS) 2012-2020 and the Medium-term Educational Development Program (MTEDP) for 2012-2014, which aim to expand the duration and coverage of the preparatory program and to increase access to lower cost models of pre-school education for children aged 4-5 years. The MTEDP also calls for the development of models for sharing the financing of pre-school education.

Finally, the Project is also consistent with the third pillar of the Interim Strategy Note (2011-13), which promotes social stabilization through social services, community infrastructure, and employment. It is closely linked to the attainment of the Millennium Development Goal (MDG) for universal completion of primary education.

b. Relevance of Design:

Relevance of Design: High

The design was based on an analysis of student performance, experience from the initial Catalytic Fund Grant, and discussions with the other partners participating in the education sector. The design built on both existing strengths and identified needs. The introduction nationally of the Government’s 100-hour pre-school education program under FTI 1 marked a shift from a previously urban-focused approach to the entire cohort of children six years of age, including those in under-served areas. FTI 2 was expected to build on the experience to expand the 100-hour program to 240 hours (i.e., a half-time program of about 3 months).
Activities were closely aligned with development objectives; the bulk of project funds (78%) were intended to introduce the preparatory program by financing the inputs needed to: (i) create a favorable learning environment for children (mostly teaching and learning materials, and classroom furniture in four of seven oblasts); and (ii) provide training, methodological support, and mentoring for teachers to improve the quality of teaching, one of the main constraints in the education system.
Finally, the Project included capacity-building activities for MoES to support the program by strengthening established practices, available staff, and physical facilities within the established school network to avoid the creation of practices and institutions for which there were no budgets, posts, or legal recognition. Key to the institutionalization were: (i) policy development to improve the management of resources necessary to achieve its mission; and (ii) monitoring and evaluation.


4. Achievement of Objectives (Efficacy) :

The Project Appraisal Document included a Results Framework with target values. The ICR includes baseline, target, and actually achieved values for both the PDO and Intermediate Outcome indicators, although the sources of information are not indicated in the Data Sheet.

Due to the short duration of the project and the time needed to mobilize inputs, the project determined that high-level outcomes or changes in the behavior of beneficiary groups could not be achieved or measured and were to be replaced by simple and measurable output indicators.

Improving learning and teaching conditions for the mandatory national school preparation program for 5-6 year old children: High

Outputs

• Preparation of a short-term training Module 1 (“Preparing Children for School: The Specifics of Training”) and guide describing the approaches to educating 6-7 year olds children in pre-school classes.

• Preparation of a training and mentoring program (Module 2, "Methodological Training in the Workplace: Organization and Support") approved by Kyrgyz Academy of Education.
• Design of a 240-hour Pre-school Preparation Program, designed for children not already covered by pre-school educational programs, approved by the Kyrgyz Academy of Education.
• Implementation of 260 training sessions involving more than 3,253 teachers and 68 methodologists throughout the country for the first cycle of training, 3,177 for the second cycle, and 2,177 participants for the full course (and certificate awarded).
• Procurement of the standard package of teaching materials for 2451 schools for the 240 hour Pre-school Preparation Program, comprising games, reading books, visual aids and methodological guides; and school furniture for 7 oblasts (instead of the 4 planned).
• Implementation of the preparatory program in 2,135 schools, compared to the 2,133 target.

Outcomes

• Expanded coverage of the 240-hour school preparation program from 0 percent to over 50 percent of the cohort of children (6-7 years old) enrolled in Grade 1 in the 2012-2013 school year.

• Completion of the preparatory program by 55,267 children, compared to the 55,000 target.

Institutionalizing the mandatory national school preparation program for 5-6 year olds within the education system: Substantial

Outputs

• Research analyzing methodological support and proposing recommendations for: (i) the policies and practices of preschool teachers; and (ii) the 240-hour pre-school preparation program. Dissemination of the results of the analysis through a series of round table discussions.

• Development of materials and delivery of an advocacy campaign to raise awareness among stakeholders of the importance of pre-school education.
• Monitoring carried out during implementation and data available and collected by MoES.

Outcomes

• In a meeting between IEG and the project task team, the team provided information on the project’s contributions to ensure the policy, regulatory, and programmatic basis for the future through: (i) passage of a new education bill; (ii) adoption of a medium-term action plan (2012-14); and (iii) development (with UNICEF) of Early Learning Development Standards (ELDS) as the basis for regulating the quality of the future 480 hour program.
• The ICR (p. 9) notes that a survey of teachers, parents, and experts carried out in 273 schools in the four project oblasts (Jalalabad, Osh, Issyk-Kul and Talas) showed that the majority of teachers and experts rated the training very high, in both in terms of content and quality. The survey also showed that more than 90 percent of those surveyed (including parents) were satisfied with the new learning environment (school furniture and teaching materials), and that rural schools tended to have higher satisfaction levels than urban schools.


5. Efficiency:

Efficiency: Substantial

An economic and financial analysis of the Project was prepared for the PAD. Financially, the FTI-2 Grant was expected to support a national program by: (i) complementing government investment in school preparedness; and (ii) covering the financing gap in the areas of the program deemed critical to increase the child’s chances of success in the early primary grades. The ICR did not present evidence on the extent to which the project achieved this financing objective.
The ICR prepared a cost-effectiveness analysis to demonstrate the project’s relevance and the potential impact (see Annex 3). In the absence of outcome data, the analysis focused on the relative contributions of pre-primary enrollment and teacher practices (Components 1 and 2 of the project, which account for 90 percent of the cost of the project) for raising PISA scores.
First, the ICR argued that research shows that: (i) gaps in cognitive and non-cognitive skills begin increasing in the early years of life; and (ii) interventions targeting the early years can raise cognitive and socio-emotional abilities of all children, especially those from disadvantaged family environments. At the outset of the project, the Kyrgyz Republic had one of the lowest pre-primary enrollment rates in the region and internationally, with less than 20 percent of 5-6 year olds enrolled in preschools in Kyrgyz, mostly due to supply-side constraints. Furthermore, pre-primary enrolment was highly skewed towards children from the two highest income quintiles.
Then, the ICR analyzed the PISA data (2009) and showed that students who participated in pre-school scored 21 points more, on average, on the reading than those who did not. In addition, teachers’ using better practices increased student scores by 10 points on average. In a meeting between IEG and the project task team, the team clarified that both findings were significant: (i) the 21 point gain constitutes about a six-month learning gain among 15 year old students who had received pre-school education; and (ii) the 10 point gain was also an important contributing factor but not as much as attendance.
Finally, using data from the World Bank’s Public Expenditure Review (2012), the ICR used a cost effectiveness analysis between pre-primary enrollment, teacher practices, and student-teacher ratio and concluded that pre-school attendance is significantly more efficient than teacher practices.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


Relevance of project objectives is rated High; relevance of project design is rated Substantial. The project was based on sound national and international analysis of the implications of pre-school education and on existing policies and programs. The project design built on experiences from the initial Catalytic Fund Grant and close collaboration with the other partners participating in the education sector, and its planned activities were closely aligned with its development objectives.
Efficacy is rated High for improving learning and teaching conditions for the mandatory national school preparation program for 5-6 year olds and Substantial with respect to institutionalizing the preparation program for 5-6 year olds within the education system. Extensive learning materials were prepared, teachers were trained, and material conditions for learning provided a sound basis for the mandatory national school preparation program. Coverage targets were exceeded. The project contributed to the policy, regulatory, and programmatic basis for institutionalization of the program.
Efficiency is also rated Substantial. The project focused on the potential contributions of increasing pre-primary enrollment and improving teacher practices. Though the former proved to be more efficient, the combined effect of the two would undoubtedly be significant in raising future PISA scores.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:

Rating:

The ICR categorized the risk to development outcome as substantial, based on the country’s limited financial resources and tight fiscal space. FTI-2 supported several measures to reduce this risk.
First, the 240-hour preparatory program was a cost-effective alternative to preschools, albeit recognizing the potentially significant difference between the two alternatives. As a result, the grant created conditions for continuous expansion in coverage of a pre-school education program with enhanced quality.
Second, the FTI-1 grant and FTI-2 grants have provided a framework fostered a strong partnership at the country level between the Government and Development Partners in support of the Education for All (EFA) goals. Within this framework, receive grant financing from the Global Partnership for Education (GPE) over the next three years to: (i) expand the preparatory program nationally to cover a full year of preschool; (ii) finance alternative models of preschool (such as community-based kindergartens); and reduce inefficiencies in the education sector (such as moving from state kindergartens to shift-based preschool education).
As noted in the ICR (p. 7), the GPE-funded project represents a transition to a financially sustainable program based on: (i) its conformity with the Government’s reform program; (ii) the transfer of salary expenditures for the newly established kindergartens to the local authorities; and (iii) efficiency improvement measures are expected to generate recurrent cost savings within both the pre-primary and general education sector. The GPE project will be complemented by (i) a Russian Education Aid for Development (READ) grant; and (ii) a Bank-finance Sector Support for Education Reform Project; both will target improved learning in primary and secondary education.
In a meeting between IEG and the task team, the team presented several counter-arguments. First, given the financial, educational, and human resource constraints, there will be (even under the new project) a large cohort of students not covered by the program: only 5-6 year olds will be covered, and they will not receive a full-fledged pre-school program. Second, government expenditures on basic education are already 6 percent of GDP and 20 percent of the government budget, so increases in such a donor-dependent country are unclear. Third, the strategy to increase coverage and control costs involves the risks that: (i) community-based kindergartens, which have worked on a small scale, can be successfully expanded; and (ii) government commitment will be available to support the program at the end of the three-year GPE grant.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The FTI-2 preparation relied heavily on stakeholder participation, including: (i) discussions with the FTI Steering Committee on project design and review procedures; (ii) collaboration with other development partners (UNICEF, the Aga Khan Foundation, and the Asian Development Bank), based on their experience with early childhood education initiatives; and (iii) consultation with non-governmental organizations (NGOs) to collect recommendations in the areas of teacher education/training and early childhood education. The Bank took the lead in: (i) ensuring that appropriate implementation arrangements were in place (with adequate fiduciary arrangements to support implementation); and (ii) preparing the required documentation for both the GPE Board and Bank management, thereby responding to the institutional requirements of both groups in a timely and satisfactory manner.

No Quality Enhancement Review (QER) was conducted. The ICR indicates that the project was prepared efficiently: (i) the time from Project Concept Note (PCN) review to project effectiveness was about fifteen months; and (ii) the total budget was forty staff weeks and US$20,000 for travel and consultant costs.

Quality-at-Entry Rating: Satisfactory

b. Quality of supervision:

Between effectiveness in March 2011 and closing in December 2012, the task team carried out four supervision missions. According to the ICR, project documentation (including Aide-Memoires and Implementation Status Reports) provided management with sufficient information pertaining to implementation progress and realistically rated the performance of the project in terms of achievement of development objectives and project implementation. The Bank’s fiduciary team also provided useful technical assistance to the Project Implementation Unit (PIU), particularly with respect to the replacement of the international auditing firm (see Section 11b). Other challenges were similarly addressed in a punctual manner, enabling implementation to remain mostly on track.
Bank supervision was very efficient. The ICR identifies only 23 staff weeks and other expenditures of US$33,000 over the 2.5-year period.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government demonstrated a high level of commitment to the objectives of the project throughout preparation and implementation. Policies and strategies were adopted to strengthen educational development, including the Government’s Education Strategy (EDS) 2012-2020 and the Medium-Term Educational Development Program (MTEDP) for 2012-2014. A Government contribution of US$1.6 million (21% of project costs) was allocated, and the ICR does not mention any difficulties in the amounts or timing of these funds. Finally, aware of the need to consider feedback from teachers, experts, and the community, the Government supported project funding for advocacy of the pre-school initiative and assessments to determine satisfaction with the efforts made.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:

Implementation arrangements conformed to those outlined in the PAD: (i) MoES served as the principal executing agency with the Deputy Minister acting as the National Coordinator for FTI; (ii) MoES nominated a focal point/coordinator in each implementing department to ensure timely technical inputs as needed; and (iii) a PIU (with adequate staffing/equipment, and prior experience managing a Rural Education Project and the FTI-1 grant) was established for day-to-day implementation of the Project. As noted in the ICR, the PIU successfully planned, implemented and monitored the Grant’s activities, ensuring coordination and sequencing. Consulting services and goods were procured in conformity with established procedures, and appropriate levels of review and approval ensured that financial accountability and follow-up were observed. No fiduciary issues were identified during implementation. Documentation was maintained properly, and the required quarterly and annual reports were submitted in a timely manner for review.

Implementing Agency Performance Rating: Satisfactory

Overall Borrower Performance Rating: Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

Output indicators were selected to monitor and measure progress. Identifying outcome indicators was discussed during design stage but considered problematic due to the expected 18-month duration of the project and concerns about attribution.

The FTI-1 Grant had found that the existing monitoring and evaluation system could not provide comprehensive national statistical data. It established a monitoring system to allow timely assessment and comparison of planned inputs to actual outputs. The design proposed using the existing tools for data collection with appropriate measures taken by the PIU to ensure accuracy and timeliness. In addition, teachers were expected to complete enrollment and attendance records, using a standardized register; mentors would be required to check the use and accuracy of the registers and to observe the use of learning materials.

Finally, public and political awareness and satisfaction about the preparatory program would be assessed by an outside firm during final evaluation, using structured interviews and/or focus groups on changed perceptions and main messages of the awareness campaign.

b. M&E Implementation:

Output monitoring drew substantially on management information tracked regularly at school, regional and central levels. A consultant was recruited to assist the PIU to monitor outputs. Regular supervision missions and a mid-term review tracked progress and identified challenges.
The PAD noted that Early Learning Development Standards (ELDS) were being developed with support from UNICEF; their completion was expected by early 2011 and would, if available, be used to assess the outcomes of the preparatory program. There is no indication in the ICR that these were adopted in time.

a. M&E Utilization:

Periodic meetings of the Implementation Steering Committee were conducted to follow-up regularly throughout the implementation period.
According to the ICR (p. 7), regular data enabled MoES to review the success of the 240-hour program, leading it to decide on rolling out a full-year 480-hour preparatory program.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

The project was rated Category C, and no safeguard policies were triggered. No environmental management plan was necessary. The project was in full compliance with Bank safeguard policies and procedures during implementation (ICR, p. 7).

b. Fiduciary Compliance:

The PIU’s financial and procurement management was sound and effective, fully meeting Bank requirements.

Financial management: The Bank’s reassessment of the eligibility of audit firms resulted in the need for additional time to search and select eligible external firms, delaying the first year’s audit. Otherwise, there were no issues.

Procurement. Two tenders were organized (in March and August 2012) to procure a standard package of teaching materials, including games, reading books, visual aids, and methodological materials; however, due to what was characterized in the ICR as an excessive level of royalties for the copyrights to learning materials included in the package, the tenders were cancelled. Instead, with the Bank’s endorsement, a revised procurement plan provided alternative reading materials through direct contracting with the Aga Khan Foundation for its tested materials. There were no other procurement issues.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Satisfactory
 
Risk to Development Outcome:
Significant
Significant
 
Bank Performance:
Satisfactory
Satisfactory
 
Borrower Performance:
Satisfactory
Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

Lessons from the ICR:
Good mentoring between a master teacher and a junior teacher and peer learning from each other can contribute to increased teacher effectiveness in a fast-track environment.

Lessons from the ICR, adapted by IEG:
In addition to improvements in the school environment, educational innovations benefit from a combination of: (i) advocacy and facilitation to help stakeholders and parents learn more about the importance and benefits of the program; and (ii) monitoring of beneficiary satisfaction to provide feedback for the expansion of programs.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR comprised sections with some very detailed analysis and others with very brief conclusions, for instance: (i) Sections 2.1 (Preparation) and 2.2 (Implementation); and (ii) Section 3.1 where relevance of design and implementation are summarized in a sentence. M&E could have benefited from more detail, especially on implementation and utilization. No summary of the Grantee’s ICR was mentioned, but in the meeting between IEG and the task team, the team agreed to provide a copy.
Annex 2 (Outputs) and Section 3.2 (Achievement) were succinctly and clearly presented, although Component 3 should have presented more analysis.
The ICR (p. 10) errs in stating that an economic analysis of the Project was not prepared at its design stage, as Annex 8 of the PAD provides a basis for possible follow-up in the ICR. In addition, though Annex 3 of the ICR provides a nice discussion, which was further clarified in the meeting between IEG and the task team.
Section 2.5 (Post Completion) and Section 4 (Risk to Development Outcome) are somewhat inconsistent.
Section 6 (Lessons Learned) seems to make several assertions which are not based on evidence presented elsewhere in the ICR. In the meeting between IEG and the task team, the team identified the sources.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions