Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Emergency Support For Social Services


  
1. Project Data:   
ICR Review Date Posted:
11/21/2013   
Country:
Guatemala
PROJ ID:
P122370
Appraisal
Actual
Project Name:
Emergency Support For Social Services
Project Costs(US $M)
 100.0  99.9
L/C Number:
Loan/Credit (US $M)
 100.0  99.0
Sector Board:
Economic Policy
Cofinancing (US $M)
 0  0
Cofinanciers:
Board Approval Date
  12/09/2010
 
 
Closing Date
06/30/2012 12/31/2012
Sector(s):
General education sector (70%), Health (30%)
Theme(s):
Natural disaster management (100%)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Denise A. Vaillancourt
Judyth L. Twigg Christopher D. Gerrard IEGPS2

2. Project Objectives and Components:

a. Objectives:


    According to the Loan Agreement of December 16, 2011,
    The objective of the Project (was) to preserve health and education services following severe flooding and other natural disasters that took place in Guatemala between May and September 2010.

    Key performance indicators (KPIs) were to provide proxy measures of the maintenance of basic education and health care services in the 130 poorest municipalities:
    the number of teachers and Telesecundaria facilitators providing education services; and
    the number of health workers providing medical services.

    While the objectives did not change, the KPI targets were adjusted (at a December 4, 2012 restructuring) on the basis of more accurate baseline data acquired during project implementation; and two new KPIs (tracking total wages paid to health and education workers in the 130 municipalities) were added to support evidence on the progress of the project development objective (PDO).

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

Component 1: Preserving Basic Education Services for Vulnerable Children (Appraisal estimate: US$70 million; Actual cost: US$80 million): This component aimed to support the Government’s Recovery Plan by preserving basic education (Grades 1 through 9) services for approximately 2.8 million vulnerable children living in the poorest 130 municipalities (out of a total of 334 municipalities in the country). It would do so by financing the salaries of qualified basic education teachers working there, as well as the salaries of Telesecundaria (distance secondary education) facilitators in these rural and isolated areas. This was expected to help the Ministry of Education (MINEDUC) to sustain education coverage as other budgetary resources were devoted to infrastructure rebuilding after the natural disasters of 2010.

Component 2. Preserving Coverage of Health Care Services for Vulnerable Groups (Appraisal estimate: US$29.65 million; Actual cost: US$19.65 million): This component aimed to support the Recovery Plan by preserving the three levels of health care and emergency services (primary, secondary, and tertiary) in the 130 poorest municipalities. It would do so by financing the salaries of qualified health care workers working there. This support was expected to help the Ministry of Public Health and Social Assistance (MPHSA) to keep health care and emergency services in these poorest municipalities of the country open and able to continue providing services, particularly maternal-infant health care for the most vulnerable groups.

Component 3. Project Management and Monitoring (Appraisal estimate: US$0.1 million; Actual cost: US$0.1 million): This component was to finance consultants’ services to support the Ministry of Finance (MoF) in the coordination, administration, financial management, audit, monitoring, and review of the project.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates

Project Cost
The actual cost of the project was US$100 million (US$99.9 million rounded up), as originally estimated. Component 1 (Basic Education Services) cost $10 million more than estimated, while Component 2 (Health Services) cost $10 million less than estimated. This shift in the use of project resources was due to (1) an overestimation at appraisal of the salary costs of health workers in the 130 municipalities; and (2) the slower than expected absorptive capacity of MPHSA.

Financing
The loan was essentially fully utilized, with US$99.9 million disbursed out of the loan amount of US$100 million.

Borrower Contribution
As planned, there was no Borrower contribution. The Borrower did, however, pay an IBRD front-end fee of US$250,000.

Dates
On May 25, 2012, under the first Level 2 restructuring, the loan closing date was extended from June 30, 2012 to December 31, 2012. This extension was made at the request of Government, following delays in Congressional approval of the loan. The loan was finally approved by the Guatemalan Congress on November 24, 2011 and became effective on December 22, 2011, almost one year after the projected effectiveness date of January 3, 2011. The net effect of these delays was that the project implementation period was reduced from 18 months to 12 months.

A second Level 2 restructuring, approved on December 4, 2012, introduced two changes. First, changes to the results indicators included: (a) a revision of the baseline and target values of the original KPIs to reflect more accurate baseline data, generated from improved Government systems; and (b) the addition of new indicators to track the monthly wage bill for education and health workers in the 130 municipalities. Second, US$10 million of the remaining loan amount was reallocated from Component 2 (salaries of health workers) to Component 1 (salaries of teachers and facilitators) to ensure the full disbursement of the loan in the 2012 calendar year. This allocation was made because the monthly burn rate for health care worker salaries was lower than expected. The health salaries were overestimated at appraisal and, in addition, the original project time period was reduced from 18 to 12 months, due to delayed effectiveness.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

High. Given the tight fiscal position that the country was facing in the wake of the severe natural disasters, the relevance of the objectives to maintain educational and health expenditures was highly appropriate and fully supportive of Pillar II of the Government of Guatemala's (GoG’s) 2010 Recovery Plan: to preserve the provision of basic services so that all, particularly the most vulnerable, would remain covered during the recovery.

The project remains highly relevant to current country circumstances, the Government’s current development priorities, and the Bank's current Country Partnership Strategy (CPS) for the Republic of Guatemala, 2013-2016. Guatemala has some of the worst indicators for health and education in the region, with important inequities across municipalities and between indigenous and non-indigenous populations. It spends a lower percentage of its GDP on these sectors than elsewhere in the region. Therefore, preserving these expenditures, which were already low, in light of an even tighter budget, was appropriate and is still a high priority. The Perez Molina administration was elected on a platform based on five pillars, two of which are focused on Social Development. The CPS supports two strategic objectives to: (1) strengthen public policies for social development; and (ii) promote inclusive and sustainable economic growth. Under the first, “Result Area 2” focuses on improving results in the social sectors, including improvements to the effectiveness of social expenditures. An emphasis is placed on ensuring the sustainability of social sector achievements.

b. Relevance of Design:

Substantial. The emergency nature of the loan caused the project’s results chain to be simple. The objective of maintaining basic services was to be achieved by ensuring that key education and health personnel in the 130 poorest municipalities would remain present on the job and continue delivering services; their continued presence was to be achieved by ensuring that their salaries would continue to be paid, during a time when substantial amounts of the GoG’s already limited budget would be used for post-disaster reconstruction and recovery. The tracking of the number of key personnel who remained on the job was used as a proxy indicator for the maintenance of social services in those areas. This indicator was relied upon in the absence of service level data that would have provided a better measure of the continued provision of services. The addition, during project restructuring, of new indicators to track wages paid to key staff improved the evidence base for tracking progress. Salaries are by far the most important and expensive input to social services provision, and an appropriate line item for the project to focus on. Other essential supplies and inputs to social services were not financed, but there were ongoing health and education projects that were poised to provide these. The focus on the 130 poorest municipalities was appropriate, given that they were already underfinanced and most vulnerable to budget cuts.

The choice of the Emergency Recovery Loan (ERL) instrument was the quickest response the Bank could offer to the GoG and made available more financing than what a restructuring of the ongoing project portfolio would have generated. The project design was kept relatively simple to ensure fast disbursement, given its emergency nature. As an emergency loan, it allowed for up to 40 percent retroactive financing.


4. Achievement of Objectives (Efficacy) :


1. Maintain Basic Education Services (grades 1 through 9) in the 130 Poorest Municipalities: Substantially Achieved

Outputs

The wages paid to teachers and Telesecundaria facilitators providing education services in the 130 municipalities rose from 115,935,000 Qtz in October 2010 to 139,606,000 Qtz in November 2012, exceeding the new target, added at restructuring, of maintaining baseline levels.

Between October 2010 and November 2012, the average monthly salary for educators in these municipalities rose from 3,404 Qtz to 4,066 Qtz. This was due to teachers unions’ success in negotiating a significant salary raise during this period.

There is evidence that the poorest municipalities received priority in the Ministry’s budget, as the share of the education wage bill devoted to these 130 municipalities rose from 15.2 percent in 2010 to 17.4 percent in 2012. During this same period, the Ministry of Education’s budget declined from 18.9 percent of total spending to 17.3 percent in 2012.

Outcomes

The number of teachers and Telesecundaria facilitators providing education services in the 130 poorest municipalities increased slightly between October 2010 and November 2012, from 34,057 to 34,334, achieving the target of maintaining baseline levels. The original 2010 baseline estimate documented in the November 2010 Emergency Project Paper (37,982) was found to be inaccurate during project implementation, and subsequently adjusted in 2012 to reflect more reliable Government data that had since become available.

2. Maintain the Three Levels of Health Care Service in the 130 Poorest Municipalities: Substantially Achieved

Outputs

The wages paid to health workers providing medical services in the 130 municipalities rose from 4,790,371 Qtz in October 2010 to 7,492,000 Qtz in September 2012, exceeding the new target, added at restructuring, of maintaining baseline levels.

Between October 2010 and September 2012 the average monthly salary for health workers in these municipalities rose from 2,952 Qtz to 4,541 Qtz.

There is evidence that the poorest municipalities received priority in the Ministry’s budget, as the share of the health wage bill devoted to the 130 municipalities rose from 1.6 percent in 2010 to 2.1 percent in 2012. While this shows success in maintaining expenditure for the 130 poorest municipalities, it also highlights the rather small portion of Ministerial expenditures reaching the poorest municipalities. This reflects the concentration of health care spending in the main cities (especially Guatemala City), home to large hospitals. The centralized nature of the health care system might also play a part, especially the lack of clear lines of reporting and payment. Health care workers based in one municipality may sometimes work in others, making it difficult to fully capture those that are working within a particular municipality for a particular month.

Outcomes

The number of health workers providing medical services in the 130 poorest municipalities increased slightly between October 2010 and September 2012, from 1,623 to 1,650, achieving the target of maintaining baseline levels. This maintenance of baseline levels was achieved after an initial decline of 6 percent between October 2010 and December 2011. The original 2010 baseline estimate documented in the November 2010 Emergency Project Paper (1,915) was found to be inaccurate during project implementation, and subsequently adjusted in 2012 to reflect more reliable Government data that had since become available.

5. Efficiency:

The project does not lend itself to standard efficiency analysis, as it is an emergency loan, which reimbursed wage expenditures for health and education. The recent Guatemala Public Expenditure Review (PER), which assessed health and education expenditures, included a standard incidence analysis of health and education programs revealing that they are pro-poor, generating important returns to poor populations. The PER concluded that Guatemala should sustain public services to improve human capital in the form of health and education, particularly for the poor, in order to reduce poverty and ensure economic growth.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:


The PDO is highly relevant to current country circumstances, GoG development priorities, and the Bank’s strategy. The relevance of the design is substantial, based on its simple, straightforward results chain, supportive of the PDO, and the appropriate choice of lending instrument. The PDO was substantially achieved, as the number of health and education workers providing services in the poorest municipalities increased slightly over the project period. Efficiency was not rated, as this was an emergency loan reimbursing wage expenditures.

a. Outcome Rating: Satisfactory

7. Rationale for Risk to Development Outcome Rating:

There is limited political will to increase social spending, despite prior increases during the period between 1995 and 2011. Increasing or even maintaining health and education services to the poorest municipalities faces several obstacles. First, Guatemala’s small tax intake means that overall public expenditures are limited. Efforts to increase tax revenues and expand the fiscal envelope have been met with strong resistance. Second, nearly 88 percent of the budget is earmarked by law, which creates significant budget rigidities and limited scope for change. Third, resource allocation tends to be based on the previous year’s allocation, rather than on actual needs, which exacerbates the historically unequal distribution of resources. Although efforts are under way to address this issue, poor municipalities will likely continue to be underserved and could be placed at risk as they also lack political clout to demand more resources.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The project design was appropriate to the emergency nature of the support, and the timeliness of the Bank’s response was commendable. The operation was prepared in only six weeks. As a consequence, there was little time to fully verify (a) the baseline data provided by Government, and (b) the ability of the Government’s systems to report monthly on the indicators. Furthermore, the salary estimates for health sector workers were overestimated. These were both detected and corrected during project implementation, when more accurate baseline data were generated and a series of reliable data from improved Government systems was produced. Given the emergency nature of the support, the Bank’s rapid response weighs more favorably than the trade-off in fully vetted baseline figures, especially given that these issues were rectified during supervision without impeding the achievement of the PDO and the project was fully disbursed.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

The supervision team worked closely with the GoG to fully disburse the project within the shortened project period (reduced from 18 to 12 months due to delays in effectiveness). The team was proactive in working with the GoG to push for regular and accurate reporting of data and to reconcile the baseline data from the Project Document with GoG systems. This ultimately involved a restructuring, with a new set of more accurate baseline data, the consequent resetting of targets (without changing the PDO), the addition of output indicators to better track progress, and the production of a series of data sets that allowed more reliable measurement of achievements.

The team also worked closely with the Bank’s health and education teams managing the two investment projects, facilitating coordination and synergies in efforts. The assurance of the payment of staff salaries was beneficial for the successful implementation of the sector investment projects.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The Government achieved the PDOs by maintaining the number of education and health care workers in the 130 municipalities. Salary expenditures were also maintained and even raised, though the number of educators and health care workers remained flat. MoE and MSPAS worked closely with the Office of Public Credit to provide the data needed to monitor compliance. The loan was prepared under extreme time pressures to meet the GoG’s request, but then it languished in Congress for almost a year before being approved.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

The Office of Public Credit/MoF (the implementing agency, responsible for project management, coordination and oversight) worked closely with MoE and MSPAS, especially on revising the integrated financial management system to produce the monitoring reports. During the life of the project, the implementing agency had to contend with changes in leadership and technical staff in the MSPAS, which had to be re-engaged and integrated into the process.

During the preparation of the loan, adequate documentation for the baseline indicators was not given. Moreover, data were not provided on a monthly basis due to the inflexibility of systems to separate service delivery staff salaries from other facility expenses. A series of four sets of data was ultimately provided, but with significant delay in response to the Bank’s request for such data series. This happened despite intensive efforts on the part of the Office of Public Credit to obtain the information from the two ministries. Government information systems were ultimately modified to facilitate the preparation and submission of the data reports.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The Office of Public Credit was charged with coordinating with the MINEDUC and MSPAS to collect the data needed for the monitoring indicators, as well as to submit statements of expenses and show payment of teacher and health care worker salaries.

The monitoring arrangements were meant to be simple, namely the tracking of the number of teachers and health care workers employed in the poorest 130 municipalities. This was expected to be done through the GoG’s financial management (SIAF) and payroll (Guatenominas) systems, which would capture the administrative data.

The short preparation time did not allow the baseline figures for the results indicators to be adequately verified. Baseline figures were drawn from information provided by GoG (and signed off by the Ministers) under significant time pressure and not backed up by data that were adequately documented and could be easily replicated.

b. M&E Implementation:

During implementation the baseline figures could not be reconciled with figures the government was providing. In addition, GoG systems were not set up to provide the data as needed (wage data for teachers, facilitators and health care workers, net of the totality of facility expenditures, including administration). As the project did not reimburse salaries of administrative staff, information on service delivery salaries/staff had to be separated from payments made to the schools and health facilities. Because of the complexity involved in producing these reports, it was not possible to provide them on a monthly basis. Instead, only four months of trend data were produced during the life of the project: October 2010 (the revised baseline); December 2011; September 2012; and November 2012. This was deemed sufficient by the Bank to ascertain that health and education wages were being paid and maintained. This consistent series of indicators for monitoring purposes took considerable time to produce, despite intensive efforts on the part of the implementing agency (the Office of Public Credit) to obtain the information from the two line ministries.

On the basis of these above-cited efforts to generate more accurate information, especially the revised baseline, the project was restructured to reset both baseline and targets. The revised baseline figures for education and health were, respectively, 10 and 15 percent lower than the original baseline figures. Targets to maintain the baseline values were revised to reflect the updated baseline figures.

a. M&E Utilization:

The Office of Public Credit initiated a modification of the GoG information systems to facilitate the provision of the data reports. The ICR does not assess the extent to which GoG utilizes the data, other than its use in evaluating this project.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

The safeguard on Environmental Assessment (OP 4.01) was triggered as the Project supported existing small health centers that produce some health care waste that should be properly disposed. This project is classified as Category C.

The Indigenous Peoples Policy (OP 4.10) was triggered as the health centers and schools supported are located in municipalities where indigenous peoples reside. The project’s support to the maintenance of existing services was expected to benefit indigenous peoples in their respective communities. Under OP 4.10, the Borrower must ensure that these benefits are culturally appropriate by undertaking a social assessment and addressing any issues or challenges identified. Because the poorest municipalities were predominantly indigenous, no separate IPP was required.

The project was compliant with the safeguards, and no issues arose during implementation. Because this was an emergency operation (OP 8.00), preparation, consultation, and disclosure of safeguard instruments were carried out after Board approval. In fulfillment of a condition of effectiveness, the Operational Manual included activities ensuring indigenous peoples’ access to services and an Environmental Management Framework. The Environmental Action Plan was published on the GoG’s website in November 2011. In addition, no safeguards issues have been raised in either of the ongoing health and education projects (Maternal Infant Health And Nutrition Project; Education Quality and Secondary Education Project), which largely covered the same poorest municipalities as the ERL and provided the basis for this project’s safeguards framework.

Under the overarching environmental safeguards for the health and education projects: a Plan for Solid Waste Disposal of Hospitals was published and served as the plan for the ERL; all public health facilities are familiar with the standards and practices for hospital solid waste management; an audit in 43 hospitals promoted compliance with standards; 110 municipalities benefited from capacity building and now have plans for solid waste management; and 36 second-tier health service points have benefited from new facilities for temporary storage of waste.

Under the overarching indigenous peoples safeguards for the health and education projects: (a) in education: by the end of 2011 there were 172 bilingual and 70 multicultural primary schools; in November 2011, 2,543 teachers graduated with qualifications in bilingual and multicultural teaching; and (b) in health: improvements in maternal-infant health services and interventions aimed at reducing chronic child malnutrition have benefited primarily indigenous municipalities.

b. Fiduciary Compliance:

Procurement. Because the payment of salaries is a recurrent cost, no procurement processes were foreseen for this significant portion of total project financing. Selection of consultants to support project administration and financial management under Component 3 were expected to follow the Bank’s Consultant Guidelines. No procurement of goods, works, and non-consulting services were foreseen under the project. The ICR does not report on compliance with Procurement guidelines. The task team noted that anticipated consultants to support project management and oversight were ultimately not recruited, as GoG considered itself to be managing well on its own. Bank guidelines for the recruitment of an audit firm were duly followed.

Financial Management. MoF was responsible for maintaining a financial management system to consolidate financial reporting and audit arrangements. Its responsibilities included: (a) the submission to the Bank of end-of-semester, interim unaudited financial reports; and (b) the commissioning and submission to the Bank of audits of Financial Statements, each covering one fiscal year of the Borrower. The ICR does not report on compliance or on whether or not audits were qualified. In a follow-up interview, the task team noted that financial management was simple and straightforward, given the one-year implementation period and the project's design of reimbursement with project funds, based on documentation of salary payments. Only one audit was required, conducted after the project closed, to cover that one full fiscal year of the project. There have been delays in the completion of the final audit report, which has not yet been received by the Bank. Follow-up discussions indicate that there seem to be no major issues raised in this report.

c. Unintended Impacts (positive or negative):

Information requirements caused MoF to make changes to its public financial systems and work closely with the MSPAS and MINEDUC to provide the needed data. Interviews with the Directorate of Public Credit commented that this was the first time they had to work so closely with the two line ministries, which proved to be beneficial for building ties between the agencies. Moreover, the technical skills gained through adjustments to the reporting system will facilitate any future requirements for customized reports.

Because of the delay in loan effectiveness and the extension of the closing date, GoG was obliged to maintain payments of service delivery salaries for longer than expected. Additionally, because of the delay in effectiveness, access to the loan in late 2011 (vs. late 2010) greatly eased the fiscal constraints at a time when the fiscal situation had reached its critical stage.

MSPAS interviews revealed that payment of health worker salaries alleviated pressures to cut other critical expenditures for service delivery (supplies, materials). In the absence of this project, such cutbacks may have taken place, which would have negatively affected service delivery.

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Satisfactory
The PDO is highly relevant to current country circumstances, GoG development priorities and the Bank’s strategy. The relevance of the design is substantial, based on its simple, straightforward results chain, supportive of the PDO, and the appropriate choice of lending instrument. The PDO was substantially achieved. While the 6-week preparation time did not permit the full vetting of baseline data submitted by GoG, the emergency nature of the operation justifies the rapid preparation, followed by the introduction of more accurate baseline values and consequent adjustments to targets, during implementation. Likewise, the adjustments to GoG systems to facilitate reporting were, in this case, appropriate for implementation. 
Risk to Development Outcome:
Moderate
Significant
Small tax intake causes limited public expenditures, and there is strong resistance to tax increases. Eighty-eight percent of the budget is earmarked by law, making budgets rigid with limited scope for change. Allocations based on previous year’s allocations (vs. needs) exacerbates the historically unequal distribution of resources. Lack of political clout of poor municipalities will likely result in continued inadequate service and underfinancing.  
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The following lesson is suggested by IEG, drawing on the findings and analysis of the ICR:

During preparation, an assessment of the capacity of a country’s reporting systems is likely to result in the choice of monitoring indicators and reporting requirements that the systems are capable of responding to. On the other hand, the choice of key indicators and reporting requirements that are beyond the capacity of systems to accommodate can result in capacity building during implementation, both of relevant staff and of the systems themselves. In the case of this project, the GoG strengthened its capacity to generate more reliable data on the number and wages of service staff, improved its systems in this light, and enhanced coordination between line ministries and the MoF through the process.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR is clear, concise, and candid in its analysis and conclusions. For the most part it covers all essential requirements. The only exception is its failure to report (in Section 2.4 “Safeguard and Fiduciary Compliance” ) on fiduciary compliance (adherence to procurement guidelines, specifically the Bank’s guidelines for the hiring of consultants; and adherence to financial management, reporting and audit requirements).

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions