Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Support For The Second Phase Of The Expansion Of The Program Of Conditional Transfers-Familias en Acción Project


  
1. Project Data:   
ICR Review Date Posted:
01/23/2013   
Country:
Colombia
PROJ ID:
P101211
Appraisal
Actual
Project Name:
Support For The Second Phase Of The Expansion Of The Program Of Conditional Transfers-Familias en Acción Project
Project Costs(US $M)
 776.2  2548.6
L/C Number:
L7619
Loan/Credit (US $M)
 636.5  636.5
Sector Board:
Social Protection
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
 
 
 
Closing Date
Sector(s):
Other social services (50%), Health (33%), Primary education (9%), Secondary education (8%)
Theme(s):
Social safety nets (50% - P) Education for all (33% - P) Nutrition and food security (17%)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Elena Bardasi
George T. K. Pitman Navin Girishankar IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The project is the fourth World Bank Loan to assist Colombia’s Conditional Cash Transfer Program (Familias en Acción) that aimed to transfer cash to poor households conditional upon them making pre-specified investments in the human capital of their children. Familias was initially implemented in 2001 to mitigate the effect of the economic downturn in the late 1990s on the consumption and wellbeing of poor households, with the objective of preserving human capital formation in poor families.

    According to the Loan Agreement (p. 5), the project’s development objectives are:
      “to support the Borrower’s implementation of its Program with a view to complementing the income of poor families with minor children under 18 years of age, promoting the human capital formation of poor children and strengthening the Program’s quality.”

    According to the Project Appraisal Document (PAD, page 6) the development objectives are to:
      “(a) complement the income of poor families with children; (b) promote human capital formation of poor children by increasing regular check-ups, for growth monitoring and other services, and by increasing enrollment and school attendance (basic and/or secondary education); and (c) strengthen program quality.”

    This Review uses the PAD’s description of objectives as they are more comprehensive and evaluable.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

The project has one component that was not revised at restructuring:

    Consolidation and Expansion of the Familias Program. (appraisal estimate, US$731.2 million; actual US$2,153.3 million). This included two main activities (Loan Agreement, p. 5):

      (a) the carrying out of a series of activities aimed at expanding and consolidating the coverage of the program to reach an increased number of poor beneficiaries with children below 18 years of age in urban and rural participating municipalities through the provision of:
      1. education grants for beneficiaries with children aged between 7 and 17 years old enrolled and attending classes in primary (grades 2-5) or secondary (grades 6-11) schools no less than 80 percent of the time; and
      2. health and nutritional grants for beneficiaries with children aged between zero (0) and six (6) years old participating in the vaccinations and growth monitoring controls according to the Ministry of Social Protection protocols and children aged between seven (7) and eleven (11) years old;

      (b) the strengthening of the technical capabilities of the Borrower’s institutions responsible for the implementation of the program through the provision of consultants’ services.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost:
The project cost increased by 400% because in late 2008 the Government decided to further expand the coverage of Familias. Between end of 2008 and end of 2009 the coverage of the program increased from 1.7 million families at the time of project appraisal to 2.7 million families. The families prioritized for the expansion were the victims of fraudulent ponzi schemes whom the Government aimed to compensate. The Government financed this expansion with national resources. In September 2011, US$554,419 was reallocated from Part1(b) (Consultant’s services) to Part1(a) (“Grants”). At that date the loan was fully disbursed and only USD 752,585 remained to be documented for the Designated Account.

Financing:
The Bank provided a Loan of US$636.50 million and this was fully disbursed. In addition the Familias Program was supported by parallel financing from the Inter-American Development Bank (IADB). The estimated financing from the Inter-American Development Bank was not included in the PAD because the figures had not been negotiated; according to the ICR, the actual IADB financing over 2009-2011 was US$251.5 million (ICR, Annex 1, p. 22).

Disbursements from the Loan began in early 2009 and were substantially completed by the end of 2010. In 2011, the main source of external financing for the program was the IADB.

Borrower Contribution:
At appraisal it was expected that the Government would provide US$139.7 million. In 2008 the Government decided to expand the coverage of the program from 1.7 million families at the time of project appraisal to 2.7 million families by the end of 2009 using its own resources. By project completion the Government contribution had increased to US$1,451.3 million and this was supplemented by US$209.3 million from the Colombian Institute for Family Welfare. The overall total was US$1,660.6 million or almost twelve times the appraisal estimate.

Dates:
The loan was extended until December 31, 2011, due to delays in the implementation of the impact evaluation of the Program in urban areas and to allow time to expense the small amount of remaining resources under the Grant Category.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

High.
Project objectives were and remain relevant to social and human development conditions in Colombia. At the time of appraisal in 2008, Colombia had benefited from sustained economic growth and a reduction in poverty for 5 years. Between 2002 and 2006, the poverty rate had fallen from 55% to 45%, and the extreme poverty rate had dropped from 20 to 12 percent. Even so, regional, ethnic and gender disparities remained, particularly in rural areas and among female-headed households, indigenous people, Afro-Colombians, and displaced individuals. Nearly 30% of the indigenous population and 25% of Afro-Colombians were considered extremely poor. A national priority was assisting needy families to invest in the human capital of their children, particularly in education and health. Although school attendance had increased across all age groups, Colombia still lagged behind Latin American averages. In rural areas, only 87% of children aged 7-11 attended school while rates dropped to 63% for 12-17 year-olds. Sizeable gaps also existed between poorer and richer departments. According to the 2005 National Demographic and Health Survey, 10% of children under 5 in rural areas and 6% of those in urban areas suffered from global malnutrition (weight for age). There were also significant differences in malnutrition rates between children in households in the lowest-ranked income group (11.6%) and those in the highest income group (3.2%).

Project objectives are highly relevant to the Government development priorities, as expressed in both the Colombia’s National Development Plan 2006-2010 (at the time of project design) and the new National Development Plan “Prosperity for All 2010-2014”. In the National Development Plan 2006-2010, Familias was a key element within the social promotion component of the social protection system and in the strategy to reduce extreme poverty – the Social Protection Network to Overcome Extreme Poverty (Juntos, now called Unidos). The National Plan for Social Prosperity 2010-14 fully supports the “Unidos” strategy, which aims to improve the coordination and strengthening of the Familias en Acción and Juntos programs, helping ensure the alignment of these two programs with other social services and income-generation and active labor market programs.During project implementation, the Government decided to expand the program to reach 2.7 million Colombian families (from 1.7 million planned at the time of project appraisal) and to finance the expansion with national resources, which demonstrated that Familias is highly relevant to Colombia's development priorities.

Project objectives were also highly relevant to the World Bank Group’s Country Partnership Strategy (CPS) for FY08-11 that was based on Colombia’s National Development Plan 2006-2010, which included five areas of concentration and collaboration (equitable growth, environment and natural resources management, good governance, poverty alleviation and equity of opportunity and peace). In the latter two areas, Familias and complementary non-lending technical assistance were key contributions to the eradication of extreme poverty, the reduction in regional inequality, strengthening the social safety net, and removing access barriers to education and health services in high conflict zones.

Projects objectives are highly relevant to the World Bank Group’s Country Partnership Strategy (CPS) for FY12-16 that selectively supports the new National Development Plan “Prosperity for All 2010–2014”. The three CPS strategic themes are expanding opportunities for social prosperity, sustainable growth with enhanced climate resilience, and inclusive growth with enhanced productivity. Under the first area, Familias is identified as a key element in the social protection system, within the social promotion component, which together with the Juntos program, has significantly improved human capital development and opportunities among poor families. One of the priorities set up by the most recent CPS is – coherently with the National Development Plan – to improve coordination between the two programs and merge them into a single program (Unidos). The CPS stressed the importance of continuity with successful operations through second and third-generation projects. The project also contributed to Colombia’s efforts to meet Millennium Development Goals and its strategy to reduce extreme poverty (Juntos).

b. Relevance of Design:

Substantial.
The project’s focus on targeting the poorest and marginal groups and their access to education and health facilities was relevant. The project design reflects lessons learned on conditional cash transfers in Colombia and other countries. In particular, project design was informed by the results of a series of previous impact evaluations in Colombia’s rural areas that highlighted the importance of improving beneficiary selection, refining educational conditionality and the importance of access to health facilities:

Improved selection of beneficiaries. The program used the SISBEN (the Selection System of Beneficiaries of Social Programs) as the main targeting instrument, which has been improved over time and had been updated before the start of implementation. Moreover, the program adopted SIPOD, the registry for displaced families, as a targeting tool for displaced household, as well as indigenous community censuses to target indigenous households.

Refinement of education conditions. Previous evaluations of Familias had shown that the program increased attendance rates for secondary school by 7% in rural and 5% in urban areas (municipal capital in municipalities with population less than 100,000). For primary school (where baseline attendance rates are higher), the impact was still positive, but reduced. The program also had a similar positive impact in both urban and rural areas in reducing the number of years of lag due to both starting school late and repetition. Education conditions have been modified in this phase of the program for cities in order to focus on the area of most potential impact, the secondary level.

Retaining condition on visiting health services. Previous evaluations of Familias had shown that the program increased birth weight and improved nutrition levels of children which would increase projected adult earnings capacity. The program also had been shown to increase preventive health visits, vaccination coverage and reduce the incidence of diseases. There were also positive impacts on consumption, particularly in rural areas, reflected in higher food (proteins) consumption, and in additional spending on children. Conditions on visiting health services were therefore be retained in this project.

The inclusion of the impact evaluation of the program in large cities (previous impact evaluation results were valid for rural areas and small municipalities) was relevant to unambiguously assigning causality from the cash transfers to outcomes in that context, and enabling the testing of the performance of three alternative structures of transfers and conditions.


4. Achievement of Objectives (Efficacy) :


The development objectives were to: (a) complement the income of poor families with children; (b) promote human capital formation of poor children by increasing regular check-ups, for growth monitoring and other services, and by increasing enrollment and school attendance (basic and/or secondary education); and (c) strengthen program quality. Achievement of each of these three objectives is presented below.

The outcomes discussed below cannot be solely attributed to the World Bank because the loan was part of a larger program of assistance to the Familias en Acción program which included parallel financing from the IADB. In addition, staff of both institutions working on the Program did not distinguish which loan was disbursing at the time. Finally, Familias en Acción was also supported by substantial Government resources and its coverage was expanded entirely with national resources.

Two different methodologies were used to measure impacts of the project: (i) impact evaluations in two large cities (Bogota and Medellin); (ii) comparison of indicators included in the administrative data for the targeting instrument (SISBEN) at two points in time (2006 and 2009 for education):

(i) The impact evaluations were based on surveys carried out in 2007 (in Medellin; early 2008 for Bogota) and February/March 2011 (endline). The main goal of the impact evaluations was to test three different structures of transfers and conditions in urban areas: in two schemes (“incremental schemes”) the value of the subsidy increased for higher grades; in the other scheme (“saving scheme”) families received a “bonus” for completing certain grades (ninth and eleventh). The overall amount received by the family was the same in all three schemes – only the structure and timing of the payment differed. The impact evaluations adopted a quasi-experimental design, where the control group was comprised of families who were not eligible to participate in the program because their scores on the targeting instrument were just above the eligibility cut-off, while the treatment group was comprised of those participants whose scores were just below the cut-off point. The impact evaluations are the main source of impacts on outcomes for objectives (a) and (b), with the exception indicated below.

(ii) The use of administrative data (i.e. a census of the participants in the programs) was necessary because the sample sizes of the impact evaluations were too small to calculate the impact of the program on some indicators, particularly in the case of education, where results were expected only for secondary students, for transition points (9th and 11th grade graduation), and for relatively rare occurrences (school dropout). Using a strategy similar to the one of the impact evaluations, the impacts derived from administrative data were calculated comparing all households eligible to participate in Familias en Acción who were just below the cut-off for eligibility with all households in Level 2 of SISBEN right about the cut-off for eligibility. This means that, as in the case of the impact evaluations, those reported in the ICR are localized impacts, that is, calculated for households around the eligibility threshold. Since international evidence suggests that the impacts of conditional cash transfer programs tend to be larger for poorer participants, the results reported in the ICR are likely to be lower-bound estimates.

The impacts reported are average impacts for the whole group of beneficiaries included in the band around the cut-off threshold used for the evaluation. That is, the average impact is calculated for the group including both program participants receiving the benefits as well as those who dropped out. However, those who remain in the program experience much higher impacts than those who do not take up the program (despite being eligible) or drop out. Overall, slightly more than 60% of targeted families benefit from the program, while 40% do not take up the program despite being eligible.
    (a) Complement the income of poor families with children: Substantial.

    Outputs:
    The total share of families who actually received their payment out of those complying with their responsibilities increased from 94.6% at baseline to 98% at endline. This percentage was similar by type of municipality at endline (95% in small municipalities, 98.1% in large municipalities, and 97.6% in urban centers), but no disaggregated figures are reported at baseline.

    Indigenous families registered in the program were 78,161 as of the end of 2011, of which 71,997 were receiving benefits during 2011. This was higher than the original target (70,000).

    The number of families receiving their payments through a bank account was 2.3 million (or more than 90% of the households in the program) as of the end of 2010 (ICR, p. 28). The program is planning to extend the opening of accounts to all beneficiaries, especially those living in remote municipalities. The program is also offering training to mothers to improve their financial literacy and incentivize savings.

    Percentage of municipalities and families verifying conditions by type of mechanism. At baseline, the main mechanism in use was presentation by mothers of evidence of compliance. At endline, the program was using to a greater extent other methods of verification including comparisons of data bases by municipal representative of program and entry of information on compliance, but precise data are not available (ICR, p. vii).

    The number of families removed from the program increased from 23,959 in 2008 to 153,092 in 2011, indicating according to the ICR (p. 29) a tighter control over the beneficiary registry due to a number of improvements in the Management Information System (MIS).

    Outcomes:
    The results of the impact evaluation show that the Program accounted for US$20 total higher spending per household per month and US$6.75 higher food expenditure per household per month between 2009 and 2011 (difference between treatment and control groups).

    The ICR (p. 12) notes that absolute levels of spending declined for both control and treatment groups (probably due to the 2008/09 global crisis), but the drop was significantly less for the treatment group.

    Familias lowered extreme poverty rates (as measured by income) by 7.5 percentage points (p.p.) in large cities compared with control groups (ICR, p. 16).

    The percentage of children consuming milk, poultry and meat was 12 p.p. higher in the treatment than in the control group and children in the Program consumed these products for 0.27 additional days per week on average. Similarly, the difference for fruit and vegetables intake was +16 p.p. and 0.49 additional days per week on average.

    The results for this objective are only reported for large cities, based on the analysis of the impact evaluation. Previous studies indicate that impacts in large cities are smaller than elsewhere, so the figures reported are plausibly lower bounds.


    (b) Promoting the human capital formation of poor children: Substantial

    The percentage of beneficiaries having received training from the program decreased from 47.5% in 2008 to 38.5% in 2011, due to the fact that the training events were suspended before elections, i.e. for reasons outside of the control of the program (ICR p. vi).

    The Program improved health:

    Children in the program were more likely than those in the control group to use preventive health services (+11.2 p.p.) with respect to the control group), and to have completed Diphtheria, Pertussis (whooping cough) and Tetanus (DPT) vaccination (+4.2 p.p.).

    Height for age among children in the Program was 0.215 standard deviations higher than among children in the control group; the percentage of children in the 'normal' range of the height for age measure was 8.4 p.p. higher in the treatment than in the control group.

    The ICR (pages 14, 34) also mentions that the program reduced the prevalence of acute respiratory disease among young children of beneficiary families but does not provide any figure, either at baseline or endline. Figures provided by the Task Team show that the project decreased the incidence of acute respiratory disease by 5 p.p. in cities outside Bogota.

    The percentage of beneficiaries reporting difficulties with the health verification process was low at baseline (1.7%) and remained low at endline (0.4%). Same for the percentage of beneficiaries reporting difficulties with the education verification process (from 1.1% it increased up to 4.4%).

    The Program improved education:

    The estimated impact on school attendance varied from +2 p.p. for boys in the savings scheme in Bogota to 13.5 p.p. for boys in the incremental scheme, according to the SISBEN census methodology. These are increases in the percentage of students attending at least 80% of the school days over 2 months (as required by the education condition). With the exception of Bogota, results were higher for boys than for girls.

    According to the same methodology, program participation increased years of schooling attained of 0.18 years in the case of Bogota, up to 0.5 years in the case of the incremental scheme - with higher results for the 11-12 year-old group (not separately reported).

    The share of students with timely school progression also increased (6 p.p. in the case of Bogota, up to 10.3 p.p. for the incremental scheme).

    The 9th grade graduation rates increased (of 6 p.p. in the case of the savings scheme in both Bogota and other cities up to 9.2 p.p. in the incremental scheme). This same indicator shows also a positive change based on the impact evaluation results, but much smaller (a 0.9 p.p. difference between treatment and control), and only driven by the positive increase of the incremental scheme (+1.3 p.p.) when the average is broken down by type of transfer scheme.

    No impacts could be estimated for high school dropout or 11th grade graduation rates.

    (c) Strengthening the Program’s quality: Modest

    Several indicators remained stable or deteriorated with respect to the baseline; a few underwent small improvements:

    The take-up rate or share of eligible families participating in the program remained virtually unchanged at 61.6% compared to a baseline figure of 62.1%. The ICR (page 13) did not set any target, but offers some elements to define a potential counterfactual. Reasons why a drop in take-up rates could have been expected were: (i) a rapid expansion of the program by nearly 60%; (ii) an expansion of the program into urban areas (with an expected lower take-up rate, as per other Latin America experience). Elements working in the direction of an improvement of the take-up rate were instead the improvements in the registration process. A move toward a more continuous registration process, which could have improved take up rates, was however postponed. The ICR states that because of the factors working in the direction of lowering the take-up rate, keeping the take-up constant was a reasonable performance. there were also geographic differences: the take-up rate decreased in smaller municipalities, from 68.0% to 63.6%, while in larger municipalities it increased from 53.2% to 55.6%.

    The take-up for indigenous people dropped substantially from 90% to 66.9% (the ICR does not offer any explanation of how this could have been the case. The Task Team explained during the interview that the two figures are not comparable - the 90% at baseline is derived from a small pilot, while the 66.9% at endline is calculated from the monitoring and evaluation data and refers to the whole group of potentially eligible people.) Elsewhere the ICR states that an evaluation carried out between August 2009 and April 2010 found that the program had effectively expanded among indigenous people in a culturally appropriate way (ICR, p. 7).
    Despite the take-up remained low, the targeting was good (in 2008 about 82% of beneficiaries were in the two lowest income quintiles). The expansion of the program to 1 million extra families and the removal of families from the program because of better screening could have affected the take-up and targeting rates, but the ICR does not provide elements to measure their effects.

    The percentage of families who comply with the health conditions sharply declined from 91.7% at baseline to 70.8% at endline, most of it occurring during 2011, for unclear reasons (ICR, p. 14). The compliance rate at endline (2011) was higher in small municipalities (79.3%), lower in large municipalities (66.3%) and even lower in urban centers (56.5%). There is no disaggregated information at baseline, but the ICR reports disaggregated compliance rates in 2009, showing that large cities accounted for the sharpest reduction, as well as - to some extent - large municipalities, while in small municipalities there was actually a slight increase in compliance (see ICR, Table 2.3, p. 30) No data are reported disaggregated by age or sex. The Task Team explained during the interview that the sharp decline in compliance with the health conditions between baseline and endline may be due to two main reasons. First, the program was extended to large cities where a lower compliance was expected. Not only may it be more difficult and more expensive to comply with the conditionality in large cities, but households may report to different health centers at different times, so verification and recording of compliance is more imprecise. Second, the implementing agency and the Bank may have become ‘stricter’ over time in defining and recording the condition as really met.

    The percentage of children who comply with education conditions dropped from 71.9% at baseline to 68.0% at endline, with a sharp reduction during 2009-2010 and some recovery during 2011. The compliance rates disaggregated by type of municipality show that, at endline, they are larger in small municipalities (68.0%), followed by large municipalities (64.6%) and large cities (58.2%). Comparisons between compliance in 2009 (not baseline) and 2011 indicate a decline in all types of municipalities, with the sharpest decline in large cities. (see ICR, Table 2.4, p. 32). No data are reported disaggregated by age or sex.

    The number of complaints was 22,871 in 2009, peaked to 73,432 in 2010 and decreased to 35,094 in 2011. No value is provided at baseline in 2008. The ICR (page vi) does not provide any interpretation of this trend.

    The percentage of complaints addressed in less than 15 working days was 92.6% in 2010 (ICR p. 31). No benchmark is reported.

    The number of municipalities with indicators of control process in yellow or red alert remained stable (from 121 in 2008 to 120 in 2011) [A municipality is considered in “yellow alert” when its performance is more than one standard deviation below the national average, and in “red alert” when it falls below 2 standard deviations. The system is used to trigger an alarm and promptly leads to the further analysis of the situation and the identification of remedial measures.]

    The number of municipalities with indicators of payment process in yellow and red alert. This indicator was not collected by the MIS. The ICR indicates that the program identifies instead, for each payment cycle, localities with higher or lower than average changes in the value of payments and sought explanations for the trend (ICR, p. 30 and p. vi), but no numbers are reported.

    The registration process improved in some aspects. In the cities, a new web system was developed to replace the old manual registration process. This helped reduce the average registration time (but the ICR does not provide data). Registration, however, remains problematic. At appraisal, the goal was to move towards a more continuous registration process (instead of periodic “one shot” events) in order to raise participation rates, but this plan was postponed (ICR, p. 13). The ICR indicates that the registration process needs to be re-thought to incorporate more out-reach and to make it easier for poor families to join the Program. A study on registration rates in five urban areas indicates that registration to the program is highly sensitive to how proactive the local government is in providing information and facilitating the registration process, among other things (ICR p. 28). Additional data shared by the Task Team show that registration to the project is expensive, averaging 6.3 hours - essentially a working day lost. However, the main reason why households do not take up the program is because of lack of information (see next point).

    Knowledge of the eligibility rules appears to have increased over time, but lack of clarity about the benefits of the program also increased. As a result, the take up did not increase between baseline and endline (information reported in the detailed impact evaluation report shared by the Task Team).

    The payment process improved thanks to the introduction of savings accounts. The ICR (p. 28-29) indicates that the bancarization process is going to continue in the future and it is expected to substantially reduce the time beneficiaries spend waiting to get paid in banks, eliminate the refund of non-collected transfers to the treasury, reduce risks in the financial process, and improve security of the payment process. The ICR does not provide data on any of these dimensions, either at baseline or endline.

    A broader set of mechanisms was used to verify compliance to conditions (in addition to presentation by mothers of evidence of compliance), including mass verification, certification cards, vouchers, smart cards, and comparisons of data bases by municipal representative of program and entry of information on compliance, although no data are provided (ICR, p. vii and 26). In commenting on the ICR, the Borrower notices that the verification process needs to be streamlined and improved, including by unifying mechanisms in order to reduce the time and cost for mothers (ICR p. 40).


5. Efficiency:

Modest.
Project-specific economic analysis to estimate the impact of the program on human capital accumulation and poverty reduction was not carried out during appraisal. Instead, the PAD reviewed the results of the existing impact evaluation of Familias, including a cost-benefit analysis, and indicated that positive impacts from the program were expected especially after modifying the transfer structure to focus on improvements in secondary (as opposed to primary) education following the extension of the program to urban areas.

The ICR (page 16) states that, because of the "substantial impact in education, concentrated as expected at the secondary school level [...], positive impacts in reducing malnutrition and the prevalence of acute respiratory disease among the young children of beneficiary families and in increasing the utilization of health services as well as vaccination coverage [...] it seems likely that the conclusion that the program has a positive benefit cost ratio would generally hold for large cities even in the absence of precise calculation".

Increased efficiency in conditional cash transfer program, though, is not only measured by a positive impact of the program on human capital accumulation and poverty reduction, but also in a decrease of transaction costs to collect benefits, and in an improvement of the targeting to reach the intended beneficiaries (the poor). The take-up rate (one of the outcome indicators) was low at the start of the program (60%) and did not improve during the program. Despite this, the targeting was good (see Achievement of objective (c) above) and because the program was expanded to 1 million extra families with respect to the number planned at appraisal one would expect that many more poor households were reached at the end of the project.

The ICR reports on innovations introduced by the program that are likely to have increased efficiency, such as the use of savings accounts to improve the payment process; the improvement in the registration process, which reduced the registration time; and the use of a broader set of mechanisms to verify compliance to conditions. These improvements are expected to have decreased the cost of the program for each beneficiary reached, although evidence of this is not provided in the ICR.

Administrative efficiency was good overall. A procurement problem arose from a combination of Bank's and client's mistakes, but was satisfactorily resolved at minimal cost.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The project had high relevance of objectives and the relevance of the design was substantial. The first two objectives (Complementing the income of poor families with children and Promoting the human capital formation of poor children) were substantially achieved, while the rating of the third objective (Strengthening the program's quality) was modest. Efficiency was also rated modest.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The Government of Colombia decided in late 2008 to further expand the coverage of Familias, beyond what was envisaged at the time of appraisal. As a result the coverage of the program increased from 1.7 million families to 2.7 families by the end of 2009. The Government financed this expansion with national resources, thus showing its commitment to the program. The new Government has indicated its intention to maintain the program, even considering a small expansion. The program continues to play a central role in improving the safety net in Colombia and is generally well regarded by the population and local authorities.

Two important changes will affect the next phase of implementation of Familias.

  • The first is that on November 3, 2011, Decree 4155 transformed the implementing agency, Acción Social, into the Administrative Department for Social Prosperity, an important institutional change in the context of the Government’s poverty reduction strategy. Acción Social was intended to be primarily an executing agency. The new Department has broader responsibilities as a rector over various components of the social protection system, including formulating and designing policies and programs. Unlike Acción Social, the new Department has the rank of a Ministry.
  • Second, for the first time since its inception, the operation of Familias will be funded almost entirely with national resources starting in 2012, once a small balance from the IADB loan is fully disbursed, mainly for studies. There are no plans for a follow-on World Bank loan to finance Familias. Already in 2011, the share of external financing had dropped from nearly sixty percent in 2010 to only seven percent. In the past some observers had expressed concern with the high share of external financing of the program; however, this does not seem to have prevented the Colombian authorities from financing it now entirely with national resources. In spite of the termination of external financing to cover the cash transfers, the program has requested to continue with the agenda of technical assistance that the Bank and IADB have jointly provided in recent years. The Bank has already held meetings with key officials in the new administration to discuss possible areas of support which would be included within ongoing programmatic knowledge and convening services.

According to the 2012-2016 CAS (p.6), the challenge is now represented by the need of further integrating the social protection system as a whole, in order to improve its reach and effectiveness. This also requires enhanced coordination between the Familias en Acción and the Juntos program, as well as between Familias and Juntos and other social services to effectively increase coverage of internally displaced people.

The World Bank will continue to provide advisory services to the Government of Columbia to improve coordination and strengthening of the Familias en Acción and the Juntos programs.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:

Appraisal incorporated lessons from previous projects in the project design. The project design was highly relevant to the country's objectives in tackling poverty and inequality. The Bank contributed to the identification of key objectives and a sound causal chain. The impact evaluation was incorporated into the project at the design stage and was to be used to test different structures of benefits in urban areas and to evaluate their results.

No targets were set for the key objectives, or for the intermediate outcomes/outputs. While this may not be a problem per se, targets on intermediate outcomes - in particular on the quality of the program - may have helped in keeping the focus on achieving improvements on outreach, take-up, and quality, especially as the program underwent a rapid expansion unforeseen at appraisal.

Quality-at-Entry Rating: Satisfactory

b. Quality of supervision:

Timely supervision missions (six in total) were carried out by the Bank team, and implementation status reports were prepared regularly. All supervision missions were undertaken in coordination with the IADB. Regular financial management supervision was carried out also. Supervision reports brought to management's attention critical issues such as the decision of the Government to expand the program, difficulties with the bidding process for banking services, the design and timing of the impact evaluation, and the need for improved coordination arrangements between Juntos and Familias. Information system monitoring focused on the indicators (both outcome and intermediate) related to the operation of the program, but this is because the performance of most of the outcome indicators (especially of objective #1 and 2) were derived from the impact evaluation. At the request of the Borrower, the Bank devoted considerable resources to providing advice and recommendations on how to better integrate Juntos and Familias to help bring about a more sustainable and effective social promotion strategy.

There was an oversight in the procurement process for banking services to handle the payment of the cash grants and open bank accounts for program participants. The Bank wrongly provided the no-objection to the bidding document, although the estimated size of the contract would have required otherwise. The Bank also underestimated the complications involved with the bidding process. The bidding lots had been grouped in a manner that had not generated the necessary competition to comply with the Bank's Procurement Guidelines, so that only one bidder presented a proposal and the cost associated with this proposal was substantially higher than expected. The ICR recognizes that the Bank should have enlisted specialist advice earlier in the process and carried out a market study before starting the bidding process.

There was a similar shortcoming in project fiduciary management during implementation of the Colombia's Social Safety Net Project, which closed in December 31, 2008 (see IEG Project Performance Assessment Report of the Colombia Social Safety Net Project, Ln. 73370 and Ln. 74330). In that instance, contract documents with commercial banks to handle the payments to beneficiaries of Familias en Acción were not subjected to a thorough procurement review. Based on previous experience, the project team should have enlisted specialist advice earlier in the process and carried out a market study before starting the bidding process, as the ICR recognizes (p. 18).

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The objectives of Familias en Acción are closely aligned to the goals of the National Development Plan 2006-10. The Government demonstrated a high level of ownership of and commitment to the program. In 2009 the Government decided to rapidly expand the program, entirely supporting this expansion with national resources. This expansion, unforeseen at appraisal, was probably too rapid, given that, as the ICR recognizes (p. 5), it may have delayed progress on some of the quality improvements and necessitated a reformulation of the design and schedule for the impact evaluation, contributing to its delay and the need to extend the project by one year to December 31, 2011. Moreover, there were delays in the process of increasing the integration between Familias and Juntos, the centerpiece of the social promotion strategy in the updated National Development Plan.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

The implementing agency, Acción Social, was highly committed to achieve the development objectives, and the key staff were qualified and stable. Consultations were carried out with mayors and indigenous communities. Disbursement was close to target. Semi-annual reports on the program were timely and sent to the Bank. The extension of one-year was not attributed to the implementing agency, but was required to redesign and carry out the impact evaluation after the extension had been decided by the government. During the course of the program, Acción Social improved the quality of the program, in particular of the Management Information System, following the recommendation of the systems audit, and improved the coordination between the program and the education and health sector. The agency also used the monitoring information to improve the program. One shortcoming in fiduciary management was due to the agency signing the contract for banking services before receiving the no objection of the World Bank.

There were weaknesses in the communication to families of the different schemes of transfers and/or difficulties in understanding the structure of the transfers (ICR, p. 5). An evaluation of operational aspects of the Program showed that knowledge increased between the baseline and follow-up survey, but in locations where the incremental scheme was implemented only about one-third of the mothers knew that the transfer level increased according to grade level. A similar situation was found where the savings scheme was implemented.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The Program incorporates into its design an impact evaluation - as it was the case of its previous phases. In this specific instance, the impact evaluation was designed to rigorously assess the impact of the program in urban areas, where the program had more recently expanded to, and to test alternative benefit structures. The availability of an impact evaluation is not standard practice in projects, and allowed measuring the impact of the program against the plausible counterfactual of "no program" in a rigorous way.

In addition to the impact evaluation, the Monitoring and Evaluation indicators used to carry out the regular process evaluation were based on the existing Management Information System that was put in place during the previous phases of Familia en Acción.

A drawback of the M&E design was the lack of disaggregation of some of the indicators. Some of the impacts were only disaggregated by type of municipality (small and large municipalities and large cities), but not by age or other characteristics of the population (for example indigenous status) as required by the PAD. The Task Team explained that the MIS monitoring system, was not set up to generate breakdowns by the characteristics identified in the PAD. Reprogramming the MIS to provide those breakdowns would have required a considerable amount of work that the implementing agency was reluctant to commit to.

Moreover, the ICR indicates that there was no progress in the take-up rate, and comments that one reason may be the expansion of the program in large cities. There are some indicators available on the costs for the households to register (in hours lost - information provided by the TTL), but some more indicators on other costs and their evolution over time could have been collected to better monitor and understand take-up rates, (for example, the time beneficiaries spend to get paid, the percentage of mothers who work, transportation costs, etc.) These indicators could have also helped in better assessing the efficiency of the program.

b. M&E Implementation:

The impact evaluation was contracted to a local consultancy firm (Centro Nacional de Consultoria). The baseline survey was carried out in late 2007 (early 2008 in the case of Bogota) and the follow-up in February/March 2011. The implementation of the impact evaluation was affected by the decision to expand the program, and as a result suffered a delay of about a year and the methodology had to be revised. The final report was delivered at the end of 2011.

The indicators included in the monitoring framework were collected by the Strategy and Monitoring unit. The quality and completeness of the information improved over time. The ICR recognizes that the proposed detailed breakdown of some indicators was difficult to achieve in practice and was probably overly ambitious (ICR, p.6). The Task Team explained that the amount of work involved in re-programming the monitoring system to generate the desired breakdowns was not known at the design stage.

a. M&E Utilization:

The ICR indicates (page 6) that data on monitoring indicators were used to organize remedial actions and local supervision visits as necessary. No example is available. The results of the impact evaluation are being taken into account in the Government’s redesign of the Familias Program.

M&E Quality Rating: Substantial

11. Other Issues:

a. Safeguards:

The project was categorized as ‘C’ for environmental impacts under OP4.01 Environmental Assessment and an environmental management plan was not required. However, OP4.10 Indigenous Peoples was triggered.

    During implementation of the previous Social Safety Net Project (Lns. 7337 and 7433), the Program carried out a pilot in indigenous communities, including consultations on the proposed strategy. These consultations and more recent ones indicated broad community support for Familias, and identified the general criteria to select indigenous communities and addressed ethnicity issues. The pilot results fed into an Indigenous People Plan that was prepared by the Borrower before project implementation, submitted for Bank review and approval, and published by the Borrower in compliance with OP 4.10. Those results were used to modify the program to adapt to indigenous communities, ensure that indigenous communities are reached and receive culturally appropriate social and economic benefits. An Operational Manual for the indigenous population was prepared as an annex to the Operational Manual of the program.

    At the time of project appraisal, Familias had started to consult with indigenous communities in about 20 departments and had initiated preparatory work in 14 localities, and expected to register 70,000 indigenous families with 152,000 eligible children during the life of the project. As of the end of 2011, the program had registered 78,161 indigenous families, of which 71,997 families received benefits during 2011. As previously reported, an evaluation carried out between August 2009 and April 2010 found that the program had effectively expanded among indigenous people in a culturally appropriate way (ICR, p. 7). New indigenous beneficiaries were selected for registration by indigenous assemblies using a process of self-targeting coherent with the criteria of the program. The assemblies were also places to resolve complaints and appeals and to oversee the implementation of the program. The evaluation found no evidence of negative consequences for the indigenous communities due to the program, but rather that the program contributed to strength indigenous institutions and empower female household head through their increased participation in the assemblies.

    No program impacts have been calculated separately for the indigenous communities, except for take-up rates.

b. Fiduciary Compliance:

The ICR states that the "implementing agency (DAPR-Acción Social-FIP) complied with the Bank’s fiduciary requirements in the area of financial management. Throughout implementation, financial management performance was rated “moderately satisfactory,… and annual financial audits were acceptable to the Bank and delivered in a timely manner. In the case of the latter, opinions were unqualified." (ICR, page 8). Moreover, a systems audit was carried out at the request of the Bank and IADB, which provided useful advice to the implementing agency on how to improve the robustness and reliability of the MIS.

There were difficulties with the procurement process for banking services to handle the payment of the cash grants and open bank accounts for program participants. The Borrower negotiated and proceeded to sign the contract with the only bidder for banking services before receiving the no-objection from the Bank.

c. Unintended Impacts (positive or negative):

The ICR mentions one positive implication from the experience accumulated by the National Coordination of the Program over the years of implementation of Familias, namely its role as participant and host of numerous south-south exchanges on Conditional Cash Transfer programs (ICR, p. 17)

d. Other:
None



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
Most indicators related to the objective to strengthen the Program’s quality remained stable or deteriorated with respect to the baseline; only a few underwent small improvements. Moreover, efficiency is rated modest because the ICR does not provide evidence on cost-effectiveness of the program (for example, costs to beneficiaries in accessing the cash transfer, and in accessing the services - health and education). The persistently low take-up seems to suggest that there may be non-negligible costs at least for some groups.  
Risk to Development Outcome:
Negligible to Low
Negligible to Low
 
Bank Performance:
Moderately Satisfactory
Moderately Satisfactory
 
Borrower Performance:
Satisfactory
Moderately Satisfactory
The Government undertook a rapid and major expansion, without taking the necessary measures to ensure quality was maintained and increased as per original plan. As a result, progress on some of the quality improvements was delayed. The implementing agency performance in informing the beneficiaries about the benefit of the program was generally poor. 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

Lessons 1 and 2 come from the ICR, lesson 3 comes from IEG.

    1. Integration of different data sources is crucial to fully assess impacts. The impacts of the program were derived from both an impact evaluation in large cities, which was meant to test the effects of alternative incentive structures, and the monitoring information system. The use of administrative data for the targeting instrument (SISBEN) at two points in time allowed the evaluators to calculate the impact for the whole group of beneficiaries and also to supplement the baseline and endline datasets in the large cities, when it became evident that the insufficient sample size of the large cities impact evaluation could not allow the estimation of some of the impacts. Moreover, the data from the monitoring information system were essential to assess progress in program quality and process outcomes. The ICR stresses that using administrative data is also cost effective (ICR, p. 21).

    2. Good public communication of program rules is important. Many beneficiaries of the transfer in large cities did not understand the benefit structure that was being tested by the impact evaluation, even two years into implementation. In addition the empowerment of indigenous assemblies was key to ensure proper information and outreach to indigenous people.

    3. Rapid expansion of a program comes at a cost and needs to be accompanied by extra measures to ensure that quality of the program is maintained. The rapid expansion of the program by more than 60% in one year (2.7 million household were reached at the end of 2009 from 1.7 million at the end of 2008) caused stress to the program and delayed to some extent the progress made on some quality improvements. The impact evaluation in large cities had to be adjusted and the project extended by one year. Moreover, this expansion delayed coordination and integration between Familias and Juntos, which was a very relevant political goal.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:


The ICR provides a comprehensive picture of project implementation and adequately discusses results. The ICR is candid in discussing challenges and shortcomings in implementation. The ICR is results-focused and the coverage of outputs and outcomes is thoroughly described with two exceptions:
    1. no elements are provided to evaluate project efficiency;
    2. impacts on outcomes are provided as percentage changes, but levels at baseline and endline are not provided for the main development outcomes, which does not allow for a full assessment of the results. (A complication is represented by having a treatment and a control group, and therefore two sets of values to report for each indicator. After the interview, the TTL provided several tables with levels. However, the ICR could have reported select key values).

The ICR was not fully clear on some of the key points, namely:
    1. The problems encountered in procurement of banking services are not clearly explained and in different places different parts of the story are emphasized (p. 8, 18, and 20);
    2. There was no information about the costs to beneficiaries (time to cash the benefits, knowledge of the program, outreach to the poor, exclusion and inclusion errors, costs to working mothers, etc.), which would have helped discussing efficiency.
    3. Some information is not substantiated with data (for example, p. 26, there is mention of a web system developed to improve the registration process and reduce the average registration time, but there is no indication of coverage or which was the registration time before and after; elsewhere, the ICR states that the "move to a more continuous registration process was postponed", p. 13).
    4. There is no mention on how successful the program was in reaching out to displaced people.
    The input of the Government was thorough and relevant.

    a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions