Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Indonesia - Bos Knowledge Improvement For Transparency And Accountability


  
1. Project Data:   
ICR Review Date Posted:
12/30/2013   
Country:
Indonesia
PROJ ID:
P107661
Appraisal
Actual
Project Name:
Indonesia - Bos Knowledge Improvement For Transparency And Accountability
Project Costs(US $M)
 2,621.5  10,190.0
L/C Number:
L7591
Loan/Credit (US $M)
 600.0  1,100.0
Sector Board:
Education
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  10/07/2008
 
 
Closing Date
12/31/2010 12/31/2012
Sector(s):
Primary education (70%), Secondary education (30%)
Theme(s):
Education for all (100% - P)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Susan Ann Caceres
Judyth L. Twigg Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Credit Agreement (p. 5), the objective of the project was "to improve access to quality education in Indonesia for all children of ages 7 to 15 by strengthening school-based management and community participation, improving existing fiduciary arrangements for greater transparency and accountability of the Bantuan Operasional Sekolah (BOS) program and consequently, bringing about better utilization of the BOS funds." The same objective was noted in the appraisal document (PAD, p. 6).

    This statement expresses a single objective (to improve access to quality education in Indonesia for all children of ages 7 to 15), as well as several means to attain this objective (strengthening school-based management and community participation, improving existing fiduciary arrangements for greater transparency and accountability of the BOS program, and bringing about better utilization of the BOS funds). For this Review, all evidence is assessed in relation to the single stated objective.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

The BOS (the acronym translates as "School Operational Assistance") program is an ongoing Government program providing block grants directly to schools for operational support. The amount of money provided to a school is based on a per-student formula. This project supported the Government's BOS program through what was essentially an on-budget transfer of funds. There was one component for this project:

    • BOS Operation Assistance School Grants, which supported the BOS grants to BOS schools (primary and junior secondary public and private schools in 33 provinces) to finance eligible expenditures as defined in the BOS Operations Manual. Funds were to be made available conditioned on compliance with mutually agreed criteria specified in the Operations Manual and progress toward achievement of the project's objectives.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost: Actual costs (US$10,190 million) were higher than originally presented in the appraisal document (US$2,621.5 million). The Government had identified a financing gap in its budget, which became the basis for the proposed loan amount. The ICR does not explain why the actual costs were so much higher than planned. However, in the Borrower's Comments, it is reported that the per-student allocation amount and the number of students covered by the program increased dramatically between 2005 and 2012 (ICR, p. 47).

Financing: The project was financed by an IBRD loan of US$ 600 million and Additional Financing of US$500 million. Bank financing was provided to the Government based on compliance with mutually agreed upon criteria in the BOS Operations Manual and progress toward achieving the development objectives. Additional financing supported the scaling up of the project and moved from strengthening fiduciary requirements to helping schools link school planning with the BOS funds. The Additional Financing allowed resources to be directed to school operational and student needs, in addition to teachers' remuneration. The Dutch Education Support Program also provided US$20 million to support the BOS program. This support focused on: (1) monitoring, evaluation, regional independent monitoring, and complaint handling; (2) social marketing and information campaigns; (3) training for BOS teams at the provincial, district, and school levels and for members of the local community; and (4) a comprehensive study of school-based management.

Borrower Contribution: The Borrower contributed US$ 9,090 million, which was 450% higher than the amount originally anticipated (US$ 2,021.5 million).

Dates: The project was effective on November 11, 2008. On June 23, 2010, additional financing was provided to the project and the closing date was extended from December 31, 2010 to December 31, 2012. The project closed two years later than originally planned.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Objectives: Substantial
At appraisal, Indonesia had made progress in access to education at the primary level, but access issues continued for poor children at the junior secondary school level. The other major shortcomings in the sector related to education quality and student learning. Indonesia passed a law in 2003 emphasizing the Government's obligation to finance basic education (primary and junior secondary). In 2005, the BOS program was initiated to provide funds to schools for operational costs so that parents would not have to contribute resources. This program is a key Government policy and is aligned with its Five-Year Medium-Term Strategic Plan promoting higher quality management of primary and junior secondary education. In 2007, regulations were updated to include school-based management and to enhance parent and community involvement.

The project's development objective reflected national goals for the BOS program, which were consistent with the Bank's focus. The current Country Partnership Strategy (2013-2015) emphasizes delivering basic services by local government and enhancing the capacity of local institutions to monitor and promote improvements in fiduciary management. The strategy also notes that progress in education access has been achieved, but that those in the poorest income quintile continue to have low enrollment. The development objectives are therefore substantially relevant to country conditions at the time of appraisal, and to the Bank's and the Government's current strategies.

b. Relevance of Design:

Design: Modest
The project supported an ongoing program of the Government, with the Bank enhancing the program through changes in the BOS Operations Manual. Design emphasized strengthening fiduciary arrangements and school management. The grants were meant to remove financial barriers for poor parents to enroll their children. Given that enrollment among the poorest quintile was lacking, removing school fees was a plausible intervention to increase enrollment. The grants also provided resources for schools to purchase materials to improve education quality. School-based management and school improvement plans were meant to increase parental and community involvement, as well as improve accountability and transparency. This would give schools authority to design and manage instruction, which in other countries has been shown to improve quality. However, other activities to improve quality were missing, even though the development objective was access to quality education, implying that weaknesses in quality would need to be addressed. Grants would more likely be effective in increasing enrollments at the junior secondary level rather than at the primary level, where near-universal education existed and more targeted interventions would most likely be needed to address factors such as cultural attitudes, disability, or distance. However, the ICR reports that the national program provided transportation subsidies to poor children, and the Government was simultaneously conducting a number of other initiatives such as building new classrooms. Thus, combining the additional Government interventions with those funded by the project may also address this design issue. The ICR does not describe additional Government programs to improve quality. Grants were provided to both private and public schools, but the ICR does not explain why both types of schools received funding.

The specific investment loan was an appropriate instrument, as it allowed funding of particular expenditures and did not support policy actions. Overall design was affirmed during a Quality Enhancement Review.


4. Achievement of Objectives (Efficacy) :

Achievements noted below cannot solely be attributed to Bank financing, but also to the Borrower's contributions and the Dutch Education Support Program (which provided technical assistance and supported studies, conferences, etc. to enhance fiduciary and school-based management).

Improve access to quality education in Indonesia for all children of ages 7 to 15: Substantial

Outputs related to the BOS Program:

The percentage of schools receiving the BOS grants on time increased from 47% in 2008 to 80% in 2012. When funding was directed to the district level, the flow slowed, but when the funds were given to the provincial level issues were resolved and schools received the resources in a timely manner (ICR, p. 16).

The percentage of schools receiving the full allocation of BOS funds based on student numbers increased from 86% in 2008 to 93% in 2012.

Annual Consolidated Reports were produced. A monitoring and information system was introduced in all schools to collect student data, and tools were also introduced to improve the handling of complaints and support transparency and accountability (such as the BOS Complaint Handling System, Tools for Reporting Information and Management Systems Team for the Acceleration of Poverty Reduction (TRIMS), and Support My School). The BOS Complaint Handling System allows community, school, and Ministry personnel at all levels to provide feedback to promote transparency and accountability. TRIMS is an evidence-based education policy instrument, introduced nationally to all schools in 2011 as part of education management information systems reforms, to support reliable and timely data collection and reporting (ICR, p. 22). Support My School is an online system for the public to report and record critical needs within schools and universities, so that these needs can be systematically resolved (ICR, p. 22). In addition, a more streamlined approach was introduced to investigate allegations of fund misuse (ICR, p. 32). There was no evidence of sanctions being applied for serious non-compliance (ICR, p. 33). TRIMS has reduced the time it takes to collect and report information from six months to less than two (ICR, p. 35). Support My School has been accessed by 4,293 users, and 1,000 unique needs have been reported. The ICR does not report how many of these unique needs have been addressed, but states that the Ministry is slowly beginning to regularly use the system (ICR, p. 35).

Socialization campaigns were conducted and materials were distributed such as leaflets, posters, calendars, magazines, newspapers, and a website. The ICR reports that posters and pocket books were distributed to every Ministry school, as well as to local districts and at the provincial level.

Outputs related to School-Based Management:

There was evidence that some parents became more knowledgeable of BOS, its uses, school budgets, and plans over the project period. The percentage of parents who knew the unit costs of BOS increased slightly from 4.4.% in 2009 to 8.2% in 2012, not meeting the target (20%). The ICR indicates that this is an indication of the need for further awareness-building activities. 58% of parents understood the objectives of BOS, meeting the target (52%). In 2012 the Ministry launched another awareness campaign and provided districts with posters.

The percentage of school committees that approved annual budget plans increased from 2.8% in 2008 to 73% in 2012. Independent monitoring suggested that most committees participated in planning and budget decision making (ICR, p. 16).

The percentage of school committees that review and approve expenditure reports was 36% in 2010, according to independent monitoring (ICR, p. 17). Data collected by the Ministry of Education based on a smaller sample of schools indicated that 87% of school committees reviewed the reports (ICR, p. 17). The percentage of schools with procurement plans was 66% according to independent survey monitoring and 71% based on Ministry of Education data. While the ICR (p. 17) states that this was an indication of increased management capacity at the schools, there were no comparable trend data provided to support this assertion.

The percentage of committees that participated in the school budget process increased from 67% in 2008 to 90% in 2011, based on preliminary independent monitoring data (ICR, p. 32). The ICR (p. 17) notes that the BOS Operations Manual was revised in 2010 to require school committees to include individuals from the community as members of the committee to increase transparency and improve community involvement. The ICR (p. 32) reports that there was limited evidence that the school committees had a substantial role in planning support for poor students in schools.

The percentage of schools with BOS expenditures by category displayed on the school notice board slightly increased from 30% in 2009 to 34% in 2012. This was meant to increase transparency in the school-based financial management process. Purchasing responsibility became part of the School Management Team after 2011.

The percentage of schools with a School Improvement Plan (SIP) increased from 41.6% in 2009 to 63% in 2012. At the end of the project period, 71% of schools had procurement plans. Schools were required to develop a four-year plan and translate this into an annual work plan and budget. School committees were required by the BOS Operations Manual to participate in the planning and provide oversight on expenditure plans and reports.

Outcomes:

Table 1 shows minimal changes in enrollment at the primary level across the project period. The largest change (2%) was noted among the enrollment of the poorest quintile. Targets were set low in all areas except for the transition rate, which appears unrealistically high. Targets were not met for boys' enrollment, poorest quintile enrollment, and transition rate, but were met for all other indicators.

Table 1 Changes in Primary Enrollment (disaggregated by gender and income), drop out and transition rates
RateBaseline (2007) (%)Target (%)Actual (2012) (%)
Primary net enrollment (total)91.192.392.5
Primary net enrollment (boys)91.492.692.5
Primary net enrollment (girls)90.992.192.6
Primary net enrollment (poorest quintile)90.994.392.5
Drop-out rate2.42.31.6
Transition rate75.683.177.3
Source: National Social-Economic Survey (SUSENAS); ICR data sheet, p. iii

Table 2 shows substantial changes in enrollment at the junior secondary level, particularly among girls and the poorest income quintile, and also in the transition rate. Nearly all targets were met, even those that were high.

Table 2 Changes in Junior Secondary Enrollment (disaggregated by gender and income)
RateBaseline (2007) (%)Target (%)Actual (2012) (%)
Junior Secondary net enrollment (total)66.568.770.0
Junior Secondary net enrollment (boys)64.468.668.1
Junior Secondary net enrollment (girls)68.768.872.0
Junior Secondary net enrollment (poorest quintile)53.065.463.5
Drop-out rate2.92.41.8
Transition rate89.289.998.0
Source: SUSENAS and Ministry of Education; ICR data sheet, p. iii

The ICR provides no data on changes in education quality.


5. Efficiency:

The ICR estimates internal efficiency gains to the education sector due to reduced repetition rates at the primary and junior secondary level and the private returns to education. As these are not an indication of project efficiency, this Review does not report them.

The ICR (p. 36) reports that, with improved financial management and accountability at the local level, there were reduced financial leakages. The Governance and Decentralization Survey (2008) reported that 70% of funds allocated by all levels of Government reached the schools. This situation has been reported to have improved, as more recent independent monitoring indicates that now 90% of the funds disbursed by all levels of the Government are received by schools. This improvement is due to the Government's overall efforts at strengthening external and internal audit systems, combined with the project's focus on improving financial management and transparency at the school level.

The project disbursed eight months before the revised closing date.

Efficiency is rated Substantial.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


The project had substantial relevance of objectives and modest relevance of design. Efficacy is rated substantial for the single objective. Efficiency was substantial. There were large increases in junior secondary enrollment rates (overall) and disaggregated for girls, boys, and the poorest income quintile, but there were only slight changes at the primary level. No evidence is provided on quality of education.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The risks to the development outcome are related to institutional capacity, particularly at the district level, and the continued need to improve education quality. However, there are several factors that help mitigate the risk to the development outcome. The BOS program is a priority within the Government, where there is a high level of commitment to the continuation of the program. There are adequate financial resources to continue the program, as 20% of the Government's budget is devoted to the education sector. The Bank has continued working with the Government and the Ministry of Education in refining the BOS Operations Manual. Improvements have been made to the funding formula to enhance equality (remote schools now receive a higher amount given their lower student enrollment). Capacity Development Plans continue to focus on strengthening management at the district level and contain budget provisions to finance this work beyond the Trust Fund. As the Government is committed to maintaining progress in attaining the development outcomes, the risk to the development outcome is moderate.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:

The Bank and Ministry engaged in productive policy dialogue while designing the project. Trust Funds were used to conduct project preparation. The ICR reports that design was informed by similar previous programs such as a Scholarship and Grant Program, a School Improvement Grant Program, and a Community Driven Development project, but the ICR does not describe the specific lessons that were incorporated into this project. Changes were made to the BOS Operations Manual, which strengthened the financial arrangements for disbursements of the block grants to schools and clarified eligible expenditure categories. This was important, as the Manual was the basis for guiding the implementation of the BOS program. The Bank team worked effectively with other donors to coordinate support and resources, as well as tailor the project within the national program.

Preparation used assessments conducted by the Bank and other donors; however, funding for the project was solely based on the Government's identified funding gap. Risks and mitigation measures were appropriately identified. Feedback received from the Regional Operation Committee and Quality Enhancement Review were used to strengthen design specifications related to controls for monitoring and auditing and improving the fiduciary control mechanism. Discussion of the Results Framework and monitoring and evaluation included BOS management at the central, provincial and district level. The Results Framework contained some shortcomings (see the sections on Relevance of Design and M&E Design).

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

The Bank integrated its work on this project with that of other donors. With Trust Fund resources, the Bank provided technical assistance and capacity building to improve the quality and use of data. The Trust Funds also provided support for the training on the Operations Manual for school staff at all levels, as well as independent monitoring and other data collection. Training was also provided to school committees.

The BOS Operations Manual was appropriately updated each year, based on knowledge gained during implementation (for example, clarification of allowable and unallowable expenditures and capping the level of BOS funds for honoraria to 20%). The Bank supported the Government's establishment of an inter-agency working group to provide a forum for the program and its education policies. During the Additional Financing, the Bank strengthened the program's focus on school-based management. The Bank used project ratings as appropriate signals to the Government and Implementing Agency. The Bank proactively worked with the Government and an inter-ministerial work group to find solutions to resolve the issue of the flow of resources to schools, after changes in this process were made by the Government.

The Bank provided adequate resources for supervision, as five missions were conducted between August 2009 and October 2012 by a team with education, evaluation, and financial management knowledge. Continuity was maintained in the task team leader (TTL) position, as one of the TTLs during preparation also supervised the project. Supervision was coordinated with Government counterparts through the Jakarta-based team. This allowed for follow-up between supervision missions, particularly related to working with multiple ministries to resolve financial management issues noted in the BOS Audit in 2011 (see Section 11b). Errors with M&E data were not detected by the Bank team until the writing of the ICR.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government was committed to the BOS program and open to revisions in the BOS Operations Manual. Disbursements were directed to the Ministry of Finance. During the initial years of the project, disbursements flowed rapidly to schools via the provincial level. However, in 2011 implementation slowed down when the Government made changes in the way BOS funds were channeled. In this new process, the funds became part of district budgets and subject to Ministry of Home Affairs financial management procedures and Ministry of Finance-mandated budget allocation lists. This eliminated the flexibility in block grants and created confusion, as the capacity in the districts to manage the education system was not well established. The Government initiated an inter-ministerial working group to find solutions to this problem. The Government rectified the situation with a return to the provincial block grant program and formalized this procedure in a regulation. Audits suggest that further strengthening of the internal audit system is still needed.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

The Ministry of Education was the Implementing Agency and established a unit (for the first two years, the Directorate General for Management of Basic Education, and for the last two years the Directorate General of Primary and Secondary Education) to manage the project. These units worked with the Bank team to revise the BOS Operations Manual. They also coordinated work with various ministries, facilitated work with provinces and districts, and prepared reports. The Ministry of Education, with the support of the Bank and other development partners, developed a training program for school supervisors, school BOS teams, and school committee members for planning, managing, and monitoring of BOS funds. This training program helped to inform school management teams and communities of the BOS program and its objectives. A Complaints Handling system was established. The Ministry implemented recommendations from the audit; however, there is no evidence of additional staff being assigned to monitor serious non-compliance of grant misuse (ICR, p. 33). The project fully disbursed eight months before the revised closing date.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The Results Framework in the appraisal document specified data collection responsibilities and a timeline for data collection and analysis. The project's design specified appropriate outcome measures and intermediate outcome indicators to measure achievement of the objective to increase access to primary and junior secondary education, but not to measure quality of education (as drop-out and repetition rates relate to many factors). Design also called for third-party data quality audits by independent monitors who conducted surveys at the district and school level. There were also several indicators that provided process data related to the BOS program. Baseline data were not specified in the appraisal document for all of the indicators, as some of the planned data collection methods only began during the project.

b. M&E Implementation:

During restructuring, changes were made to some of the key indicators. A mistake in the wording of one indicator (drop-out rate) was corrected. Some indicators were dropped and replaced with others. The changes enhanced the indicators, as they become more measurable or attributable to the project. For example, it was wise to drop the indicator tracking average monthly household expenditure per student on education by the poorest income quintile, as this would have been difficult to measure and was related to many factors outside the control of the project.

Several process evaluations and surveys were undertaken to better understand school-based management and the school marketing information campaigns. These additional data collection activities were supported by Trust Funds. The independent survey monitoring was done within all 33 provinces and covered 2,160 schools. The Ministry of Education's research and development agency conducted monitoring and evaluation of program implementation. This was meant to identify implementation problems as encountered. The independent monitors provided timely data to the Government and the Bank. Additional analysis was undertaken by the Bank on school-based management in primary schools.

Data from national economic and social surveys (SUSENAS) were used to measure enrollment at primary and junior secondary levels. The baseline data in the ICR and the PAD are therefore different, as different rounds of the SUSENAS were used, with the Bank concluding during preparation of the ICR that a latter round was more reliable than an earlier round. Other data drawn from the Ministry of Education were used to monitor transition and dropout rates. The ICR uses the most recent data provided by the Ministry; during the preparation of the ICR, the Bank team determined that some data were not properly weighted/calculated, and so the ICR team recalculated those indicators.

a. M&E Utilization:
Data were used to inform changes in the program and monitor implementation.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

The project was rated Category C for Environmental Compliance, as the project did not finance any civil works. No other safeguard policies were triggered.

b. Fiduciary Compliance:

The Ministry of Education maintained adequate fiduciary management. The internal audit agency undertook audits of the BOS program. At the end of the project, there were no outstanding Financial Management Reports. Quarterly interim financial reports were submitted. After changes were made by the Government in how resources were channeled to schools, the National Audit Bureau issued an Audit Report with a qualified opinion (this was the final audit report for 2011, issued in 2012). This opinion showed that control systems at the district level were weak and the control system could not be fully verified. The Ministry of Education followed up on the recommendations related to the provincial and district levels. The opinion points to differences between the control systems in the Ministry of Finance and Ministry of Education. There was no evidence of fund misuse. In 2012 the Government spelled out that BOS funds would be released from the provincial levels directly to schools (in Regulation no. 62/2011) to avoid future issues.

There was limited procurement in the project. This limited procurement was implemented according to Bank policies and procedures.

c. Unintended Impacts (positive or negative):
None reported.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Satisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The ICR (pp. 25-26) provides a number of recommendations, synthesized by IEG as the following:
    • Good and reliable data at preparation are essential for monitoring and evaluating new approaches in the delivery of education services. In this case, independent monitoring and survey data monitored the process, but their value was decreased as there were no baseline data at preparation.
    • Establishing and maintaining a continuous funding mechanism is a prerequisite condition for schools to have efficient and timely grant disbursements. During this project, there were substantial delays when the funding mechanism was changed. This resulted from staff members who were poorly suited for their responsibilities, conflicting regulations, and schools that were not prepared to develop detailed annual budgets. Maintaining stability in the funding mechanism is preferred, but if changes are made, advanced capacity building for all entities is needed.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR is concise and clearly written. It describes implementation thoroughly and discusses achievement of the project's objectives primarily through assessment of the formal project indicators. The Borrower's comments were helpful as a supplement to this Review.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions