Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - (India: Elementary Education (Sarva Shiksha Abhiyan II)


  
1. Project Data:   
ICR Review Date Posted:
11/11/2013   
Country:
India
PROJ ID:
P102547
Appraisal
Actual
Project Name:
(India: Elementary Education (Sarva Shiksha Abhiyan II)
Project Costs(US $M)
 10,700  22,914
L/C Number:
C4417
Loan/Credit (US $M)
 600  1350
Sector Board:
Education
Cofinancing (US $M)
 30  595
Cofinanciers:
DfiD, EU
Board Approval Date
  06/05/2008
 
 
Closing Date
03/30/2010 09/30/2012
Sector(s):
Primary education (90%), Central government administration (10%)
Theme(s):
Education for all (100% - P)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Eluned Schweitzer
Denise A. Vaillancourt Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:

According to the legal agreement for the second Sarva Shiksa Abhiyan (SSA II) or Second Elementary Education project, dated August 14, 2008 (p.5), the objective of the project was to " support the Recipient's ongoing program for Elementary Education as described in the Sarva Shiksa Abhiyan framework for implementation, with the aim of achieving universal enrollment and completion of elementary education of satisfactory quality through significantly increasing the number of 6-14 year old children in India, especially from marginalized groups such as Scheduled Castes (SC) and Scheduled Tribes (ST) enrolled, regularly attending and completing elementary education and demonstrating basic learning skills".

The PAD's (p. 3) statement of the Project Development Objective (PDO) was shortened, but is consistent, in substance, with the legal agreement "to significantly increase the number of 6-14 year old children, especially from special focus groups, enrolled, regularly attending and completing elementary education and demonstrating basic education learning levels." The objectives were not revised but some of the targets were revised at restructuring, which was approved by the Board, along with extra financing, in March 2010.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

SSA II was the second phase of a Sector Wide Approach (SWAP) in which a consortium of donors directly supported the ongoing education program of the Government of India including budget support to the States which are responsible for implementing education programs. The broad Government program was launched in 2001-2002 to provide a comprehensive framework to achieve universal elementary education. The Government program included other National government sponsored schemes which complemented the activities under donor funding. The SSA II project activities were substantially affected by the passage of the Right To Education Act (2009) which mandated specific actions on the part of States and had a specific human rights approach. The implementation framework for SSA was rewritten after the passage of the Right to Education Act, with mandates on access and quality within specific timeframes. The donor funded project covered over one million government aided schools in all Districts and States of the country as part of the broader program.

Original Components

a) Improving Quality with Equity (Improving Educational Quality p. 5 of Legal Agreement): This component broadly included a number of activities which aimed to create enabling learning conditions for all children:

i) ensuring the basic provision of teacher training, textbooks and educational materials.
ii) capacity building and academic support to State and Sub-State structures
iii) monitoring educational outcomes
iv) research and evaluation studies

b) Universalizing Access and Promoting Equity: This component financed activities which aimed to enable the hardest-to-reach children to attend school.

i) the establishment, construction and extension of primary and upper primary schools and classrooms where need was highest
ii) the provision of free textbooks and other materials to reduce household costs for education

Components Post Re-Structuring:

The Passage of the Right to Education Law in 2009 generated additional activities and policy changes in the education sector which significantly affected the project activities. Although project components were not changed, additional activities were included after March 2010 to accommodate these requirements.

Expanded activities included: (these varied from State to State)

Component a)

-The hiring of additional subject-specific upper primary teachers
-The recruitment of additional lower primary teachers and the implementation of teacher development programs.
-The rollout of Learning Enhancement Programs (LEPs) focused on reading and numeracy in early grades in more than half the schools.
- Improved academic support to schools through Block Resource Centers (BRCs) and Cluster Resource Centers (CRCs)
- Strengthening of School Management Committees through training on management and fiduciary responsibilities and the preparation of School Development Plans.
- Support to policy changes in 8 States which converted their education systems from 7 to 8 grades.
- Assistance to States for curriculum renewal , including associated student assessment and teacher professional development.

Component b)

- Additional expansion of primary schools to elementary schools (from grades 1-5 to 1-8), construction of additional upper primary classrooms, procurement of teaching and learning equipment and classroom furniture,

- The distribution of textbooks to all upper primary students.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Costs: Project Costs at appraisal were US$10,700 million; and actual costs were US$22,914 million on March 31 2012. . No costs per component are given in this review, as only costs for itemized activities are available in the ICR.

Financing: The Second Elementary Education Project (SSA II) was designed using a Sector Wide Approach (SWAP) financing mechanism, essentially budget support mostly passed on to the State Government level. Outlays for activities were approved based on District level decentralized annual planning under the Government Education Strategy. Bank and Development Partners contributed less than 10% of the total SSA Government Program funding. The United Kingdom's Department for International Development (DfID) co-financed the project with about US$30 million including a Technical Cooperation (TC) Fund of approximately US$10 million, and later provided an additional US$220 million including an additional US$8 million for the TC Fund. The European Union (EU) provided approximately US$100 million via the SWAP mechanism over the life of the project.

An additional IDA credit of US$750 million dollars was approved in March 2010.

Borrower Contribution: Eighty five percent of education expenditures take place at the State level. According to the Bank's team, at the beginning of the project the National government was providing 75% of counterpart funding with 25% being provided by State budgets. By the end of the project, funding had reached parity between State and National government sources. There were no counterpart funding issues as the budget for these activities increased over the life of the project.

Key Dates: The project was restructured in March 2010 when an additional credit of SDR482.6 million (US$750 million) financing was approved. This additional financing covered a gap in Government of India financing for FY2009-10 following the full disbursement ahead of schedule by the original IDA credit. Without this additional financing, achievement of the project's development objectives would have been put at risk (see section on project components for additional activities financed). At closure, two and a half years later than the original date, IDA project funds were fully disbursed.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Rated High:


The Objectives of the project were relevant at the time of appraisal and remain so.

Government National Strategy: Education in India is a State responsibility within a federal system. However, the Government of India has a national program and targets which the States are required to implement. Although the first five years of the Government Education Program had seen significant increases in enrollment at the primary level, with a reduction in the number of out of school children (OOSCs), 13.5 million children still remained without access to primary school. Gender parity was being achieved and the transition rate from primary to upper primary had increased from 75% to 83%. However, challenges remained in 2007, particularly at the upper primary grades and for marginalized groups such as the Scheduled Castes (SCs) and Scheduled Tribes (STs). There were also challenges in terms of the quality of the primary education provided. This particular project supported the broader Government SSA program and its priorities and the State education programs. Revisions in 2009 expanded the project emphasis on primary grades 1-5 (ages 6-10), to include upper primary schools or grades 6-10 (ages 11-14). combining these into elementary schools which covered grades 1-8.

Bank Strategies: The 2004 Country Assistance Strategy (CAS) had supported the education sector at the elementary level through a number of previous projects. Project activities supported the achievement of several Millennium Development Goals MDGs), including universal primary education, gender equality and the empowerment of women. Donor funding (including IDA) focused on a combination of catalytic activities relating to the improvement of quality and basic assistance to improve access in terms of facilities and materials. The focus of activities changed slightly at re-structuring in 2010 but remained in essence the same.

The current Country Partnership Strategy FY2013-2017 still highlights the need for investments in elementary education: "India's efforts to improve access, equity, and quality of education at the primary, secondary and tertiary levels remains a work in progress"(p. 25). Improved access and quality of education remains a goal under the Engagement Area 3, which emphasizes inclusion, particularly for disadvantaged groups and the remaining out of school children and a third project is proposed. Although there are still gaps in access to quality education for vulnerable groups, there is a natural progression, which this project started to bridge, from lending for primary education to the upper levels of education, as enrollments and retention improve and access becomes more equitable.

b. Relevance of Design:

The Relevance of Design is rated Substantial. The project was the second phase of programmatic support for National Government of India activities at the primary and upper primary levels of education. Joint donor funding (including IDA) was intended to support National Government efforts to improve the quality and equity of the education system by encouraging States to emphasize federal goals through their State education programs. The results chain was coherent and supported the achievement of the PDOs, including essential inputs to quality and access (teacher ratio improvements, free/new textbooks and learning materials and infrastructure) and a focus on vulnerable groups and Districts in order to improve equity.

The project design and implementation mechanisms to achieve the PDOs remained the same as Phase I of the SSA program. The components were chosen to help achieve national objectives, but States had the flexibility to define activities and mobilize financing to implement them in response to local level challenges. As project activities were rolling in nature and identified at State level within the national framework for education, the open ended nature of the components was appropriate. The mechanisms for operationalizing activities were improved over the earlier SSA I design in several ways: the approach to financing from the National level to the States became more performance based, and the Annual Workplan and Budget approval process targeted resources to Districts most in need.


4. Achievement of Objectives (Efficacy) :

The particular context of this project (especially the decentralized design to give States flexibility in implementing the broad themes of a program) constrains what a typical ICR could report on in several ways: (a) the ICR does not provide State level data on outputs; and (b) this information is difficult to provide in detail in any case, given the autonomy and flexibility the States had in implementation in line with their local level needs and populations. The scope of interventions varied across States, Districts and Sub-Districts and from year to year in response to these needs. Outputs under the Project Development Objectives (PDOs) identified below can contribute to more than one outcome (provision of quality buildings contributes to quality as well as access).

Increase Enrollment of 6-14 Year Olds, with Equity: Substantially Achieved

Outputs for overall enrollment:

  • Construction of 161,600 primary schools and 97,300 upper primary schools and the addition of a total of 650,442 classrooms to existing schools. Some of these were lower level learning centers in remote areas which were upgraded to regular schools. Infrastructure resources were targeted to most in need areas. The construction component did not quite meet its targets, however, due to lack of capacity in particular States and targets were revised at re-structuring.
  • Distribution of free textbooks
  • Specific State level initiatives to attract out of school children and minority groups (further detailed under PDO II)
  • Specific quality improvement initiatives (further detailed under PDO II).

Outputs for improving enrollment equity:
  • Gender Issues: School construction included improvements in drinking water and toilets and separate toilets for girls. The percentage of schools with toilets increased from 86.75% in 2008 to 95% in 2012. The number of primary level schools with toilets increased from 40% in 2007/8 to 62% in 2012. These increases cannot all however be attributed to the project. Other special initiatives such as girls residential schools and a National Program for the Education of Girls at Elementary Level (NPEGEL) were targeted to educationally deprived areas. Student hostels were provided for remote areas and urban areas where children were at risk, and transport and escort services provided to help children get to school where access was difficult.
  • Special Needs: Children with special needs (CWSN) were identified, the number increasing from 2.63 million in 2007-08 to 3.27 million in 2012. .
  • Minority issues: A number of special strategies were used to address equity issues. Sixty one special Focus Districts (SFDs) were formed based on the number of Scheduled Castes and Scheduled Tribes and Muslim minority populations. The provision of mid day meal schemes and free textbooks as well as information campaigns would have also contributed to the increases in enrollment for deprived groups.
  • Out of School Children (OOSCs): Specific initiatives were put in place so that the absence of required paperwork would not prevent children attending school and children could be enrolled throughout the school year, as well as for children who were late enrollees. Special urban plans were put in place to try and reach OOSCs in urban areas where community links are weaker.

Outcomes for enrollment:
  • The number of out of school children dropped during the period of SSA II, although varying across age cohorts. A focus on enhancing access to upper primary levels resulted in increased enrollments in this age group. Data from a number of surveys were triangulated to judge the outcome for this development objective. These were: i) the yearly household child census; ii) third party Ministry of Human Resource Development surveys and; iii) the independent National Sample Survey. All these indicate that the target figures for enrollment have been exceeded. A new Household Survey is underway to get current data. One figure quoted from State governments for OOSC was 2-3 million for 2012 -2013.
  • The surveys also show that a large number of remaining OOSC are concentrated in a few States and Districts which account for between 40 and 57% of current OOSC. Data provided by the country education team demonstrate that about 20% of the two poorest quintile students from Scheduled Tribes were enrolled in primary and upper primary education in the year 2009-2010. About 47% of the poorest two quintiles of Scheduled Castes were enrolled at those levels during the same timeframe. Thus although access has improved enormously, there are still specific groups of the most hard to reach who have not yet been impacted.

Outcomes for enrollment equity:
  • At the time of project closure, girls as a share of students enrolled in primary and upper primary education reflected their percentage of the population. This is not entirely surprising as overall gender ratios were improving at the beginning of the project timeframe.
  • The share of SC and ST students has also increased, reaching or exceeding population share. There has been an estimated increase of 21% in the number of these children enrolled in education. This is a significant improvement leading towards fully inclusive education, particularly given the known difficulties in every country in changing attitudes towards this population
  • Although resources were meant to be targeted to Districts of Special Need it is not clear which of these targeting approaches was the most successful where and why, or what the specific effects were on enrollment in these areas. More analysis and information would clarify which approaches could or should be scaled up to address the issues for the remaining OOSC and reduce social gaps. Some identified gaps include uneven access to appropriate facilities for girls and urban migrants, and specific States where addressing the poorest remains an issue. These issues are a focus for the upcoming Education Project.
  • Students from mid to upper income levels are entering the private education sector. Calculations using nationally representative sample surveys indicate that this affects even the lowest quintiles, although the proportions are very small.
  • Overall the proportion of children (6-10 years) from the poorest quintile not attending school dropped from slightly over 20% in 2004 to about 12% in 2009-2010, a significant achievement. From quintile two, the proportion dropped from about 16% in 2004 to about 6% in 2009-2010. These data come from the ICR. However, IEG observes that there is not much overlap in the 2004-2010 trend data and the project life (2008-2012) so that caution must be used in attributing these trends solely to the project.

Increase the number of 6-14 year olds regularly attending and completing elementary education: Substantially Achieved

Outputs: Transition rates would have been impacted by a wide range of project inputs and outputs (itemized above)

Outcomes for the general population:
  • Transition rates from primary to upper primary increased from 81% to 86.6% during the target period, which was above the target rate of 85% and the objectives were essentially achieved.
  • Elementary retention (ages .to 14) also improved from 32% to 55% exceeding the target of 45% in States where the cycle was 8 years. However, the retention rate at primary level (ages 6-11) has increased only marginally and did not reach the target of 80%. Finding and targeting those children who still drop out at the end of primary would be key to improving these rates overall. It may well be that another internationally known gender issue has emerged - that of certain groups of boys dropping out at the end of primary to enter the workforce. This gender issue is mentioned on p. 71 in the ICR in the comments from the Government.
Outcomes for specific groups:
  • Elementary Completion rates for specific groups (grade 8) improved. For example between 2003 and 2010 the completion rate for Scheduled Tribes boys jumped from 35% to 63% and for girls 21% to 54%. Similarly completion rates for Scheduled Caste students went from 38% to 65% for boys and 25% to 60% for girls (National Assessment Survey data). Thus good progress was made by these groups during this time period in terms of completion, However, significant issues remain for other minority groups, again in specific areas of the country. The ICR is lacking data on these issues. It is also of note that these trends overlap only partially with the project period (2008-2012).

Increase the number of 6-14 year olds demonstrating basic learning skills: Modestly achieved

Quality-related outputs:
  • Quality improvement aspects of this program included improvements in curriculum, pedagogy, materials, classroom organization and practices, teacher recruitment, deployment and development, evaluation and research. Fifteen States revised the curriculum and school syllabus with twelve smaller States following their example. Some of these reforms are difficult and long term and require additional capacity building and support.
  • Thirty-eight percent of the 1.27 million teachers appointed in the whole SSA period were appointed under SSAII and the mean Pupil Teacher Ratio (PTR) was 30:1 in 2011-2012, down from 34:1 in 2007-08 in spite of a large student increase. However, due to regional variations, 35% of Districts and 40% of schools still have a PTR above 30:1. There is little information about the qualifications of these teachers in the ICR. Information from the ICR team indicates that a parallel government program was working with the teacher training institutions and this would be important in order to ensure that new cohorts of teachers were trained in new methodologies. In-service training and Block Resource centers were provided under the project which would have assisted teachers in improving their practice. Average teacher attendance at Primary and Upper Secondary almost met a target of 90%, an increase of 10% over the initial baseline value, and other improvements to teacher deployment and utilization were made.

Outcomes:
  • Measurement of student achievement is most commonly undertaken at national level by student assessment programs. The program of national assessments in place at project commencement were designed using Classical Testing Theory. Surveys carried out using this methodology indicated marginal improvements in grades 3,5 and 7/8. This assessment was redesigned using funding from the Technical Cooperation Fund. Only one assessment at grade 5 was implemented under SSA II which was not truly comparable to the previous test, given the difference in test approaches. However, using common test items it was possible to see that some scores improved, although these improvements were small. Again there are significant differences in performance across States and children have still in many cases not mastered basic skills. Not only subject knowledge but critical thinking skills are lacking. These results are validated by those from other independent surveys.
  • Given the time it takes to improve quality and for changes to become integrated into the system, it is not entirely surprising that learning achievement results are mixed. Variation between Districts and States may also account for much of this national level result, as well as the movement of students from the upper quintiles to the private sector.

There has been significant progress in putting in place an improved system of quality inputs and improving measurement of learning achievement, however, these have not yet fully translated into improved learning achievements as measured nationally. Further work on these issues is included in a proposed SSA III project.

5. Efficiency:

Efficiency is rated Substantial. At appraisal the project team analyzed the costs and benefits associated with the project and calculated a potential rate of return for the project. The ICR notes that there is known evidence that the private returns to primary and elementary education in developing countries such as India are higher than those of developed countries. Returns are high because completion of basic education enhances wage earning capacity and improving the quality of education further enhances this potential. There are also significant social returns to education, such as impacts on infant mortality and improved health seeking behaviors.

In fact some of the data presented in the ICR reveal that although there is a positive individual rate of return for primary education at the present time, this is diminishing as primary education becomes more available. Discussion with the ICR team confirmed that there are variations in this return between States and Districts. Increasingly the greatest return to education in India is at the secondary and higher education levels, although there are again significant variations in this regard. This would be commensurate with the economic growth of the country and improvements in access to education in recent years. The calculations use an estimation of 80% and above employment rates for project beneficiaries. However, these calculations use estimations of engagement in the formal and informal sectors for which data are uncertain, causing some concern about their reliability.

Regardless of any economic analysis, the number of children out of elementary school would in any case demand that quality basic schooling, which is considered to be a human right, be made available to all as a first step in the education system. This is reflected in the human rights focus of the Right To Education Act of the Government of India which the project supports.

The project objective of targeting resources to deprived groups such as scheduled caste and tribes, girls and children with disabilities is difficult and costly to implement as these are the hardest to reach groups, both socially and in some cases geographically. Incorporation of these millions of children into the education system is necessary for their futures, regardless of potential rate of return analysis results. And it is pivotal for reaching India's goal of universal enrollment. The targeting of efforts and resources to this end is thus efficient.

Delays in implementation caused by State level capacity problems in implementation and financial management led in some cases to inefficient use of resources. It seems that some States or Districts were unable to use allocated funds in a timely fashion because of this issue. On the other hand, transaction costs for the Government were reduced by the use of the SWAP mechanism. This enabled coordinated supervision of program activities by donor partners and the use of a common results framework as well as improved national systems for financing flows between Government and States.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
19.36%
100%
ICR estimate:
Yes
19.4%
100%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Relevance: Relevance of objectives is High and relevance of design is Substantial. The project was a good fit with country needs, donor priorities and Bank strategies and remains relevant under the current CAS. Although there were design flaws in the M&E framework (insufficient intermediate outcome indicators to adequately capture the breadth of project activities and a lack of indicators on administrative improvements ), the planning requirements for implementation and use of the M&E framework had a substantial effect on data usage for decision making in the education sector.

Efficacy: Efficacy is rated Substantial. The project helped the Government of India and State Governments further their agenda to improve basic education and target out of school children. Such a rapid expansion of staff and facilities in such a large country is bound to be a work in progress. The project did achieve its stated results in improvements in enrollment and retention with equity and set the stage for further progress in improving quality, through multiple changes to the content and delivery of elementary education services. Learning outcomes, however, fell short of project targets. The implementation of program activities at the State level contributed to greater efficiency in operationalizing nationally mandated programs, built capacity at State and local levels and greatly contributed to improved stakeholder engagement in education at the local level.

Efficiency: Efficiency is rated Substantial. The project appropriately targeted efforts and resources to most vulnerable/disadvantaged groups and Districts, essential for achieving national goals. Delays in implementation caused by State level capacity problems in implementation and financial management led in some cases to inefficient use of resources. However, investments in State capacity building are expected to generate benefits in the medium term, and transaction costs for the Government were reduced by the use of the SWAP mechanism.


a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:


Political, Institutional and Financial: The Government still strongly supports the SSA agenda under the 12th Five Year Plan and the recent legislation and institutional development mechanisms in place ensure the political and financial sustainability of the program. Administrative risks do remain in specific areas of the country where administrative capacity and accountability mechanisms are less well developed. These variations would be the case in any large Federal country. Weak fiduciary capacity in some States at decentralized levels was a risk factor at appraisal and remains an issue. Staffing program activities adequately at State level also seems to be a key ingredient needed for the long term success of a program of this scale.

Technical: More needs to be done to ensure that teaching staff are using up to date methods, that learning is measured appropriately, and that schools are held accountable for results. Without these continued inputs the enrollment rates may decline again, particularly for vulnerable populations, and retention and transition rates may well also slide. Hence the continued emphasis on quality and on reaching the most difficult to reach populations is critical for the maintenance of current improvements.

Social: There are still issues with access and quality of programs for OOSC with disabilities and from minority groups in specific areas. These tend to be the poorest in the poorest States and are the most difficult to reach.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The project design addressed issues in the 2004 CAS and government and donor priorities. The design of the project built on lessons learned in the first phase of the SSA with relevant priority activities and mechanisms for implementation. The design benefited from a Quality Enhancement Review (QER) and a Bank-wide Regional Operations Committee (ROC) review. Project preparation was adequately staffed and financed. The project used manuals developed under SSA I which enabled States to prepare annual work plans.

However, the project design overestimated the capacity of some of the States to implement project activities which caused delays in implementation and lags in achievement of project activities, particularly construction. The design also did not include any specifically identified component or theme which addressed the institutional development activities at State level (such as the institution of Annual Workplans) although some of these clearly led to improved efficiency in the delivery of education services (see section on Monitoring and Evaluation).

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

The project was closely supervised from the World Bank's India office and was also supported by the Technical Capacity fund provided by the United Kingdom Department for International Development( DfID). Implementation Status Report ratings were realistic in terms of implementation of activities and progress towards development objectives. Financial management issues were flagged and addressed by the World Bank Financial Management staff and the implementing team. A Mid-term review was carried out in January 2010. Only one formal supervision mission took place in 2011, although these increased in frequency during the latter part of the project.

The SWAP financing mechanism meant that Joint Review missions with the government and donors took place, with each agency providing supervision staff, thus minimizing the impact of many visiting supervision missions on the work of the implementing agencies.

Bank staff also contributed to policy dialogue through analytic reports and policy and technical assistance notes.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The project was a second phase of a country-wide education program financed through a SWAP mechanism. The government demonstrated a high level of commitment to the objectives of the project which had national leadership through the Chairmanship of the National Elementary Education Mission by the Prime Minister. A high level of political will for achieving project goals was demonstrated through consistent budgetary increases supported by increased taxation. Collaboration between Bank and Borrower was constructive throughout project design and implementation.

In FY2008-2009 both Central and State Government allocations and expenditures exceeded the expectations set out in the PAD and all IDA resources were expended six months prior to the original closing date. There were no counterpart funding issues. The Government demonstrated its commitment to project goals by continually increasing expenditures for primary education and working closely with States to ensure that central mandates were complied with.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:

Project activities were implemented by the Ministry of Human Resource Development (MHRD) and 35 States and Union Territories using the existing government departments, mechanisms and financing flows. These performed at different levels in different States and Districts. Several States faced implementation challenges due to limited staff capacity, particularly the availability of engineers and supervisory staff at the District and Sub District level. State Project Directors had more than one role, changed rather frequently and there was sometimes a lack of continuity in key personnel affecting the efficiency of expenditures. These constraints meant that government staff who were implementing the project could not effectively absorb increasing budget allocations which could not be disbursed on time. There was a shortage of financial management and audit specialists at local level.

The implementing agencies had an overall good record in terms of procurement, which is significant given the scope and level of purchasing undertaken with the funds.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The project framework contains a good set of macro level indicators, which reflected progress made against project objectives at the national level (and State level for enrollment). These indicators were already in use by the government and are broken down by vulnerable group and gender.

Prior to Re-structuring the project included relatively few intermediate outcome indicators to measure the effect of specific project activities. Several additional indicators were formally added to the intermediate outcome results framework at the re-structuring, but these could be considered outputs rather than measures of intermediate outcomes and are related to construction and the addition of toilet facilities for girls. Even considering the flexibility of the SWAP design in which activities differed from State to State and over the life of the project, it might have been possible to include some more specific outcome indicators, particularly for the activities relating to improving the administration and efficiency of the education system and improving stakeholder involvement at the District level such as the number of States with performance based contracts and their effect on performance, and the number of Annual Workplans and budgets and the impact of this on the distribution of resources. Similarly the number of schools using school report cards and any changes in performance that could be attributable to improved transparency would have been interesting. There is a depth of experience in a project of this size which is not easily captured by national level data.

The project design did not incorporate plans or central mandates for pilot programs which could have been scaled up and measured, or indicators for such programs. It is not clear how the independent studies, both Indian and International, which are used for project analysis feed into the improvement of project activities financed through the National government. The M&E design could have better captured the complex nature of what was undertaken through the inclusion of some further State level data or data on identified sample high risk Districts.

b. M&E Implementation:

Forty-one independent monitoring institutions reviewed progress and outcomes on SSA II. At the District level, the District Information System for Education (DISE) was used for monitoring school level indicators and school report cards were available on the internet. Data from independent agencies such as the National Sample Survey were used to triangulate results on access and equity and special studies tracked improvements in learning. However, it appears that monitoring was more effective than evaluation (ICR page 22). More efforts were needed to ensure that States could evaluate their programs and share lessons learned. Program evaluations in four States were published in the last year of the project.

The Technical Cooperation Fund financed by DfID strengthened efforts to improve national assessments and train staff to undertake these as well as the evaluation of quality initiatives.

a. M&E Utilization:

The SSA program emphasized improving data quality and its increased use in decision making. This was the case at all levels of the education system. The results framework helped States prepare annual work plans which greatly facilitated program implementation, particularly at the local level. For example, information on out of school students taken from the household census was used for community monitoring of out of school children. District level data improved significantly and was used to identify special focus Districts and an Education Development Index (EDI) to target resources according to need.

However, more effort could have been devoted to promoting and nurturing the use of data and information for decision-making at the local level, and for the sharing of lessons between States.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

An Environmental Assessment (EA) was undertaken in September 2007, including findings from National Third Party Evaluations (TPE) in several States. The project received a B rating. Environmental issues were related to civil works, specifically the construction of toilets and water facilities at schools. A combination of Joint Review Missions, periodic implementation reviews and other monitoring groups ensured compliance and were effective at identifying issues. An innovation was the inclusion of environmental audits which were conducted by special teams of engineers within the State Implementation Agencies. These covered all States in a period of three years.

b. Fiduciary Compliance:

Fiduciary compliance was identified as a risk at appraisal. The financial management system was guided by a comprehensive Manual on Financial Management and Procurement disseminated before the start of the program. Some strengths included: a) a robust flow of funds through electronic channels, which in some States extended to the District level; b) a standardized audit procedure; c) regular review meetings at State and national level; and d) the rollout of financial software.

However, there were significant issues including: a) weak internal controls; b) delays in the receipt of audit reports and in rectifying mistakes identified in audit reports; c) a shortage of up to 50% in the number of required financial management staff at local level and the continued use of manual accounting in some areas. These issues led to a rating of Moderately Unsatisfactory from early 2011 to closure and capacity building measures were unable to resolve them by project closure. These remain an issue.

On the procurement side, activities were handled by agencies at all levels of government and communities according to the Financial Management and Procurement Manual and audit and procurement reports were disclosed. Post Reviews conducted during implementation showed no major procurement issues and those issues which did arise were dealt with effectively.

c. Unintended Impacts (positive or negative):

A program called Rashtriya Madhyamik Shiksha Abhiyan (RMSA) which addresses the lack of secondary education was started for grades 9 and 10 in response to increasing transition rates to secondary education. This theme is being taken up in the third SSA project.

The RTE legislation and the SSA project focused national and international attention on education issues. As a result Civil Society and NGOs were empowered to engage in various aspects of education.

d. Other:

In spite of the above-cited issues, the program contributed overall to the development of improved fiduciary practices, including financial management and associated training which are now being adopted by the Rashtriya Madhyamik Shiksha Abhiyan (RMSA) program for two grades of secondary education with financial support from the Bank.

It also had significant impact on institutional development in terms of:
a) Improvements in planning and budgeting through the Annual Work Plans which are now integral to dialogue between State and National governments. These Annual Work Plans are a core feature not only of this project but of the education management system. The Government has issued directives to States and States to Districts on the timing and requirements of these Workplans.
b) Strengthening the capacity of School Management Committees which manage funds and hold schools accountable for education. The empowerment of School Management Committee volunteers also had a social benefit in empowering local stakeholders.
c)Streamlining and coordination of programs for school students with disabilities
d) Improvements in the capacity of the National Council of Education Research and Training (NCERT) which developed Quality Monitoring Tools (QMT) and has now developed the capacity to conduct national learning assessments to international standards.



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Significant
Significant
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The ICR mentions a number of lessons learned including:

a)" Strong political commitment and ownership are critical for successfully steering a centrally sponsored scheme. Significant and positive results - especially on the access and equity agenda - demonstrated by the program has proved to be the linchpin for sustained political ownership and commitment."

b)" SSA provided Development Partners (DPs) with a viable platform to provide consistent and concordant support to the program under a common results framework" The engagement of the DPs not only reduced transaction costs for the government but resulted in timely identification of issues and resolution of implementation challenges.

c) "A realistic assessment of capacity is important to inform the design and sequencing of the reform programs for optimal impact."

d) Learning assessments themselves are complicated to design and administer and require considerable institutional capacity to be run at scale. They are also only a technical exercise if the results are not shared with stakeholders and acted upon.

Page 79 of Annex 7 in the Government response to the ICR, notes additional lessons:

-The importance of decentralization and delegation of the planning, management and monitoring of interventions has emerged as a key lesson learned;

- The participation of civil society is a crucial factor in ensuring stakeholder participation, and

"The most crucial lesson learnt is that interventions for equity and quality have to run hand in hand and not one after the other." Unless education is of uniformly good quality, access to school alone cannot create equity between the disadvantaged and the more fortunate.

One further lesson identified by IEG, drawing on the ICR review process, is that identified areas of exceptional need, such as particular regions, States or Districts, require intensive efforts and expenditures to reach acceptable standards in service delivery and to change schooling patterns for the most vulnerable groups. Approaches to these groups may need to vary between geographic area and need specific monitoring and evaluation. Extra effort is needed to ensure that addressing the needs of these vulnerable groups is successfully achieved within a broader program as decentralization in a federal system does not automatically lead to equity.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR is clearly written, with a good overview of the project rationale and its design features. It captures the context in which the project was developed and provides a good analysis of the results against the project PDOs and outcome indicators. It provides a significant amount of analysis based on a broad range of data and focuses on results, triangulating evidence where there is no one source of information. It is consistent with IEG guidelines, concise and internally consistent.

This was a huge and complex project and it is difficult to capture the extent of what was accomplished. The counterfactual is particularly challenging to assess given the nature of the SWAP design. In- State variation in needs, inputs and results have not been fully captured. Some further examples of State level improvements might have given a better flavor of the scope of project achievements and remaining challenges.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions