(a) Complement the income of poor families with children: Substantial.
The total share of families who actually received their payment out of those complying with their responsibilities increased from 94.6% at baseline to 98% at endline. This percentage was similar by type of municipality at endline (95% in small municipalities, 98.1% in large municipalities, and 97.6% in urban centers), but no disaggregated figures are reported at baseline.
Indigenous families registered in the program were 78,161 as of the end of 2011, of which 71,997 were receiving benefits during 2011. This was higher than the original target (70,000).
The number of families receiving their payments through a bank account was 2.3 million (or more than 90% of the households in the program) as of the end of 2010 (ICR, p. 28). The program is planning to extend the opening of accounts to all beneficiaries, especially those living in remote municipalities. The program is also offering training to mothers to improve their financial literacy and incentivize savings.
Percentage of municipalities and families verifying conditions by type of mechanism. At baseline, the main mechanism in use was presentation by mothers of evidence of compliance. At endline, the program was using to a greater extent other methods of verification including comparisons of data bases by municipal representative of program and entry of information on compliance, but precise data are not available (ICR, p. vii).
The number of families removed from the program increased from 23,959 in 2008 to 153,092 in 2011, indicating according to the ICR (p. 29) a tighter control over the beneficiary registry due to a number of improvements in the Management Information System (MIS).
The results of the impact evaluation show that the Program accounted for US$20 total higher spending per household per month and US$6.75 higher food expenditure per household per month between 2009 and 2011 (difference between treatment and control groups).
The ICR (p. 12) notes that absolute levels of spending declined for both control and treatment groups (probably due to the 2008/09 global crisis), but the drop was significantly less for the treatment group.
Familias lowered extreme poverty rates (as measured by income) by 7.5 percentage points (p.p.) in large cities compared with control groups (ICR, p. 16).
The percentage of children consuming milk, poultry and meat was 12 p.p. higher in the treatment than in the control group and children in the Program consumed these products for 0.27 additional days per week on average. Similarly, the difference for fruit and vegetables intake was +16 p.p. and 0.49 additional days per week on average.
The results for this objective are only reported for large cities, based on the analysis of the impact evaluation. Previous studies indicate that impacts in large cities are smaller than elsewhere, so the figures reported are plausibly lower bounds.
(b) Promoting the human capital formation of poor children: Substantial
The percentage of beneficiaries having received training from the program decreased from 47.5% in 2008 to 38.5% in 2011, due to the fact that the training events were suspended before elections, i.e. for reasons outside of the control of the program (ICR p. vi).
The Program improved health:
Children in the program were more likely than those in the control group to use preventive health services (+11.2 p.p.) with respect to the control group), and to have completed Diphtheria, Pertussis (whooping cough) and Tetanus (DPT) vaccination (+4.2 p.p.).
Height for age among children in the Program was 0.215 standard deviations higher than among children in the control group; the percentage of children in the 'normal' range of the height for age measure was 8.4 p.p. higher in the treatment than in the control group.
The ICR (pages 14, 34) also mentions that the program reduced the prevalence of acute respiratory disease among young children of beneficiary families but does not provide any figure, either at baseline or endline. Figures provided by the Task Team show that the project decreased the incidence of acute respiratory disease by 5 p.p. in cities outside Bogota.
The percentage of beneficiaries reporting difficulties with the health verification process was low at baseline (1.7%) and remained low at endline (0.4%). Same for the percentage of beneficiaries reporting difficulties with the education verification process (from 1.1% it increased up to 4.4%).
The Program improved education:
The estimated impact on school attendance varied from +2 p.p. for boys in the savings scheme in Bogota to 13.5 p.p. for boys in the incremental scheme, according to the SISBEN census methodology. These are increases in the percentage of students attending at least 80% of the school days over 2 months (as required by the education condition). With the exception of Bogota, results were higher for boys than for girls.
According to the same methodology, program participation increased years of schooling attained of 0.18 years in the case of Bogota, up to 0.5 years in the case of the incremental scheme - with higher results for the 11-12 year-old group (not separately reported).
The share of students with timely school progression also increased (6 p.p. in the case of Bogota, up to 10.3 p.p. for the incremental scheme).
The 9th grade graduation rates increased (of 6 p.p. in the case of the savings scheme in both Bogota and other cities up to 9.2 p.p. in the incremental scheme). This same indicator shows also a positive change based on the impact evaluation results, but much smaller (a 0.9 p.p. difference between treatment and control), and only driven by the positive increase of the incremental scheme (+1.3 p.p.) when the average is broken down by type of transfer scheme.
No impacts could be estimated for high school dropout or 11th grade graduation rates.
(c) Strengthening the Program’s quality: Modest
Several indicators remained stable or deteriorated with respect to the baseline; a few underwent small improvements:
The take-up rate or share of eligible families participating in the program remained virtually unchanged at 61.6% compared to a baseline figure of 62.1%. The ICR (page 13) did not set any target, but offers some elements to define a potential counterfactual. Reasons why a drop in take-up rates could have been expected were: (i) a rapid expansion of the program by nearly 60%; (ii) an expansion of the program into urban areas (with an expected lower take-up rate, as per other Latin America experience). Elements working in the direction of an improvement of the take-up rate were instead the improvements in the registration process. A move toward a more continuous registration process, which could have improved take up rates, was however postponed. The ICR states that because of the factors working in the direction of lowering the take-up rate, keeping the take-up constant was a reasonable performance. there were also geographic differences: the take-up rate decreased in smaller municipalities, from 68.0% to 63.6%, while in larger municipalities it increased from 53.2% to 55.6%.
The take-up for indigenous people dropped substantially from 90% to 66.9% (the ICR does not offer any explanation of how this could have been the case. The Task Team explained during the interview that the two figures are not comparable - the 90% at baseline is derived from a small pilot, while the 66.9% at endline is calculated from the monitoring and evaluation data and refers to the whole group of potentially eligible people.) Elsewhere the ICR states that an evaluation carried out between August 2009 and April 2010 found that the program had effectively expanded among indigenous people in a culturally appropriate way (ICR, p. 7).
Despite the take-up remained low, the targeting was good (in 2008 about 82% of beneficiaries were in the two lowest income quintiles). The expansion of the program to 1 million extra families and the removal of families from the program because of better screening could have affected the take-up and targeting rates, but the ICR does not provide elements to measure their effects.
The percentage of families who comply with the health conditions sharply declined from 91.7% at baseline to 70.8% at endline, most of it occurring during 2011, for unclear reasons (ICR, p. 14). The compliance rate at endline (2011) was higher in small municipalities (79.3%), lower in large municipalities (66.3%) and even lower in urban centers (56.5%). There is no disaggregated information at baseline, but the ICR reports disaggregated compliance rates in 2009, showing that large cities accounted for the sharpest reduction, as well as - to some extent - large municipalities, while in small municipalities there was actually a slight increase in compliance (see ICR, Table 2.3, p. 30) No data are reported disaggregated by age or sex. The Task Team explained during the interview that the sharp decline in compliance with the health conditions between baseline and endline may be due to two main reasons. First, the program was extended to large cities where a lower compliance was expected. Not only may it be more difficult and more expensive to comply with the conditionality in large cities, but households may report to different health centers at different times, so verification and recording of compliance is more imprecise. Second, the implementing agency and the Bank may have become ‘stricter’ over time in defining and recording the condition as really met.
The percentage of children who comply with education conditions dropped from 71.9% at baseline to 68.0% at endline, with a sharp reduction during 2009-2010 and some recovery during 2011. The compliance rates disaggregated by type of municipality show that, at endline, they are larger in small municipalities (68.0%), followed by large municipalities (64.6%) and large cities (58.2%). Comparisons between compliance in 2009 (not baseline) and 2011 indicate a decline in all types of municipalities, with the sharpest decline in large cities. (see ICR, Table 2.4, p. 32). No data are reported disaggregated by age or sex.
The number of complaints was 22,871 in 2009, peaked to 73,432 in 2010 and decreased to 35,094 in 2011. No value is provided at baseline in 2008. The ICR (page vi) does not provide any interpretation of this trend.
The percentage of complaints addressed in less than 15 working days was 92.6% in 2010 (ICR p. 31). No benchmark is reported.
The number of municipalities with indicators of control process in yellow or red alert remained stable (from 121 in 2008 to 120 in 2011) [A municipality is considered in “yellow alert” when its performance is more than one standard deviation below the national average, and in “red alert” when it falls below 2 standard deviations. The system is used to trigger an alarm and promptly leads to the further analysis of the situation and the identification of remedial measures.]
The number of municipalities with indicators of payment process in yellow and red alert. This indicator was not collected by the MIS. The ICR indicates that the program identifies instead, for each payment cycle, localities with higher or lower than average changes in the value of payments and sought explanations for the trend (ICR, p. 30 and p. vi), but no numbers are reported.
The registration process improved in some aspects. In the cities, a new web system was developed to replace the old manual registration process. This helped reduce the average registration time (but the ICR does not provide data). Registration, however, remains problematic. At appraisal, the goal was to move towards a more continuous registration process (instead of periodic “one shot” events) in order to raise participation rates, but this plan was postponed (ICR, p. 13). The ICR indicates that the registration process needs to be re-thought to incorporate more out-reach and to make it easier for poor families to join the Program. A study on registration rates in five urban areas indicates that registration to the program is highly sensitive to how proactive the local government is in providing information and facilitating the registration process, among other things (ICR p. 28). Additional data shared by the Task Team show that registration to the project is expensive, averaging 6.3 hours - essentially a working day lost. However, the main reason why households do not take up the program is because of lack of information (see next point).
Knowledge of the eligibility rules appears to have increased over time, but lack of clarity about the benefits of the program also increased. As a result, the take up did not increase between baseline and endline (information reported in the detailed impact evaluation report shared by the Task Team).
The payment process improved thanks to the introduction of savings accounts. The ICR (p. 28-29) indicates that the bancarization process is going to continue in the future and it is expected to substantially reduce the time beneficiaries spend waiting to get paid in banks, eliminate the refund of non-collected transfers to the treasury, reduce risks in the financial process, and improve security of the payment process. The ICR does not provide data on any of these dimensions, either at baseline or endline.
A broader set of mechanisms was used to verify compliance to conditions (in addition to presentation by mothers of evidence of compliance), including mass verification, certification cards, vouchers, smart cards, and comparisons of data bases by municipal representative of program and entry of information on compliance, although no data are provided (ICR, p. vii and 26). In commenting on the ICR, the Borrower notices that the verification process needs to be streamlined and improved, including by unifying mechanisms in order to reduce the time and cost for mothers (ICR p. 40).