Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Sergipe State Integrated Project: Rural Poverty


  
1. Project Data:   
ICR Review Date Posted:
02/22/2014   
Country:
Brazil
PROJ ID:
P110614
Appraisal
Actual
Project Name:
Sergipe State Integrated Project: Rural Poverty
Project Costs(US $M)
 US$27.10  US$25.10
L/C Number:
L7595
Loan/Credit (US $M)
 US$20.80  US$18.49
Sector Board:
Agriculture and Rural Development
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  09/23/2008
 
 
Closing Date
12/31/2011 06/30/2012
Sector(s):
General agriculture fishing and forestry sector (55%), Water supply (15%), Power (15%), Sub-national government administration (10%), Other social services (5%)
Theme(s):
Rural services and infrastructure (29% - P) Participation and civic engagement (29% - P) Rural non-farm income generation (28% - P) Rural policies and institutions (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
John R. Eriksson
Ridley Nelson Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The PAD (pp.14. 49) begins with a statement that the Project Objective is "an integral part of the Government of Sergipe’s rural development with inclusion strategy." This phrase is a presumed characteristic of the Project Objective, but is not the objective itself. Following this phrase, the PAD sets our three specific Project Objectives.


    "The project will ....
    (i) improve the rural poor’s access to basic socioeconomic infrastructure,

    (ii) raise incomes and capital assets through investments in productive activities and increased linkages between small producers and markets, and

    (iii) strengthen the cross-sectoral integration of investments in rural areas through improved participatory planning and monitoring at the local, municipal and State levels."

The Loan Agreement, dated May 11, 2009, gives the following formulation of the project objective (p.6, Schedule 1):
    "The objective of the Project is to increase social and economic opportunities for Municipalities’ rural poor by improving their access to basic social and economic infrastructure and thus contributing to the Borrower’s objective of improving its Human Development Index."

The first part of this definition is an elaboration of the first objective in the PAD formulation ((i) above), but the Loan Agreement formulation lacks the second objective in the PAD, to "raise incomes and capital assets through investments in productive activities and increased linkages between small producers and markets." This objective is considered by the ICR to be the "core of the project" (p.14). However, the Loan Agreement formulation adds a specific objective not found in the PAD. Namely, to contribute to the “Borrower’s objective of improving its Human Development Index.” As will be explained in sections 3 and 4 below the Human Development Index was never envisioned as an objective with baselines and targets in the project, but rather as a criterion for selecting municipalities to participate in the Project. Accordingly, IEG adopts the specific objectives as formulated in the PAD.

The PAD states a third specific objective, to "strengthen the cross-sectoral integration of investments in rural areas through improved participatory planning and monitoring at the local, municipal and State levels." The ICR treats this as a project objective and assesses it. The PAD and ICR also refer to this objective as a project component, “Component 3” (“Horizontal Integration,” briefly described in section 2-c below). The participatory aspect could be viewed as an objective but the processes involved are means to achieving the first and second project objectives. IEG will assess the third objective but recognize that it is also a means.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
The Project was the second operation of a planned Bank-supported two-phase Rural Poverty Reduction Program supporting the State of Sergipe. The Federal Government divided the planned operation of US$41.6 million into two loans of US$20.8 million each, with performance conditions for moving to a second phase. Due to fiscal non-compliance issues between the Federal Government and the State of Sergipe, the State could not proceed with an Additional Financing and Phase I closed in June 2006. Consequently, there was a gap of 3.5 years between the Concept Note Review (March 2005) and Board Approval (September 2008) of Phase II and another 10 months to Project Effectiveness in July 2009. The Project implementation period, including a six-month extension, was 3 years.

The Phase II Project had four components:
1. Community Subprojects
2. Institutional Development
3. Horizontal Integration
4. Project Administration, Supervision, Monitoring and Evaluation

Component 1: Community Subprojects (appraisal: US$23.46 million; actual: $US21.86 million)
To support, through matching grants to community associations, implementation of about 1,000 subprojects for (i) small-scale socioeconomic infrastructure and (ii) productive and market-oriented investments with technical assistance (small-scale agricultural and nonagricultural related). This Component was intended to improve the livelihoods and wellbeing of an estimated 100,000 people in about 670 communities. As under the first phase project, communities, through their associations and participatory Municipal Councils, were to identify, prioritize and execute investments. Some 70% of subproject investment resources were to be focused on the 41 poorest municipalities. The remaining 30 municipalities were to receive an estimated 30% of funds for subproject investments.

Component 2: Institutional Development (appraisal: US$1.38 million; actual: $US1.05 million)
To be executed by the implementing agency, the Sustainable Development Company of the State of Sergipe, and to include: (a) technical assistance and training to support mobilization and strengthening of community associations to identify, prepare, operate and maintain subproject investments; (b) capacity-building for municipal councils to manage responsibilities defined in the Project Operational Manual, including assessment and supervision of Community associations, participatory planning, financial management of community subprojects, and environmental management of small community subprojects; and (c) workshops and seminars for municipal councils and community associations to exchange implementation experiences and facilitate integration with other federal and state poverty reduction programs.

Component 3: Horizontal Integration (appraisal: US$0.82 million; actual: --- [no expenditure])
To provide technical assistance, consultancies, software acquisition, and systems installation to the State Secretariat of Planning to strengthen its results-based management capacity to improve planning, integration, monitoring and evaluation of public policies and investments for poverty reduction; to improve the alignment of public expenditures with the State’s development priorities, and to support modernization of state public administration.

Component 4: Project Administration, Supervision, Monitoring and Evaluation (appraisal: US$0.54 million; actual: US$1.68 million)
To finance operational costs of project administration, coordination, supervision, monitoring, and impact evaluation.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost
The total actual project cost was US$24.59 million, before an IBRD Front-end Fee of US$0.52 million, which brought the total to US$25.10 million. This compares with an estimated total cost at appraisal of US$27.1 million, comprised of the sum of the costs of the four components, US$0.80 million for contingencies and US$0.05 million for a Front-end Fee (PAD, p.64).

Financing
The project was financed by an IBRD loan of US$18.49 million or 73.7% of the total cost (including front-end fee). The balance of US$6.62 million was contributed by the Borrower and Local Communities.

Borrower Contribution
The Borrower contributed US$4.43 million or 17.6% of the total cost and Local Communities, US$2.19 million or 8.7%. The Borrower's share of total cost increased from 14.1% at appraisal to 17.6% at closing (Local Communities’ share declined slightly from 8.9% at appraisal to 8.7% at closing). The increase in the Borrower's share resulted from an absolute increase in its contribution from US$3.83 at appraisal to a contribution of US$4.43 million at closing, as well as from a decline in the IBRD loan from US$20.80 million at appraisal to the final amount of US$18.49 million. The decline reflects a cancellation of US$2.31 million, owing to a refusal by the Bank to extend the closing date beyond six months (ICR, p.12). The reason for the Bank position is explained in section 5 below, on Efficiency.

Dates
The project was approved 09/23/2008 and closed 06/30/2012 after a six-month extension.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
The relevance of objectives is rated substantial.

The assessment of relevance and achievement of objectives in the next two sections focus primarily on the first two specific objectives of the project (see section 2-a above): "(i) improve the rural poor’s access to basic socioeconomic infrastructure; (ii) raise incomes and capital assets through investments in productive activities and increased linkages between small producers and markets." The relevance of these objectives is substantial. At appraisal, the State of Sergipe – the smallest State in Brazil – ranked 23rd nationally in its Human Development Index (HDI) with 54% of its population still classified as poor. Rural poverty was especially severe with 50% of rural households depending on less than one minimum salary per month (about US$280 at the time) and 61% on less than two. While aspects of socioeconomic infrastructure had improved, with government programs taking over the provision of electricity, almost half of households still lacked piped water, and jobs and income generation had, with the demands of participatory community institutions, become an urgent focus of public policy.

The Bank's Brazil Country Partnership Strategy 2012-2015 noted relevant Government priorities for addressing climate change and other environmental concerns, ameliorating slum conditions and associated poverty, and investing in sustained water resource use (pp. 9, 24, 30, 32, 126). Of the 24 Bank lending operations identified in the Indicative Lending Program for FY 2012-2013, half are focused on States in the Northeast of Brazil and most deal with poverty, rural development or environment issues (pp.44-45). Of Economic and Sector Work (ESW) underway or planned, about a third is focused on the above subjects in the Northeast (pp.47-48).

The third specific objective, “strengthen the cross-sectoral integration of investments in rural areas through improved participatory planning and monitoring at the local, municipal and State levels," is modestly relevant since it is a means to accomplish the first and second specific objectives. It could be considered substantially relevant if “cross-sectoral integration” and “participatory planning and monitoring” are considered objectives but the relationship between these constructs as means and as objectives is not made clear in the ICR.

As noted above, a specific objective in the Loan Agreement is “contributing to the Borrower’s objective of improving its Human Development Index.” The components of a Human Development Index, such as infant mortality, are related to the first specific project development objective (socioeconomic infrastructure) contained in the PAD. But no attempt was made in the ICR to establish a linkage or measure it. Moreover, the Team points out that the components of the Human Development Index change slowly over time and would therefore not be relevant for the Project, even if change could be measured.

b. Relevance of Design:
The relevance of design is rated modest. The first two Project Components --Community Subprojects and Institutional Development-- are relevant to the first and second specific project objectives.

The relevance of the third Component --Horizontal Integration-- is not clear. The PAD indicates that the Component is “to provide technical assistance, consultancies, software acquisition, and systems installation to the State Secretariat of Planning to strengthen its results-based management capacity to improve planning, integration, monitoring and evaluation of public policies and investments for poverty reduction; to improve the alignment of public expenditures with the State’s development priorities; and to support modernization of state public administration.” These activities could presumably contribute to the first and second specific project objectives, as would Component 4 (“Project Administration, Supervision, Monitoring and Evaluation”), which considerably overlaps Component 3.
The Team indicates that the purpose behind the Horizontal Integration concept was to build capacity in State Secretariats of Planning in States of the Northeast to manage integration of policies and programs across sectors for greater development efficiency and impact. Sergipe State declined to borrow for such activities but the Project Team indicates that one element of Component 3, the training of middle managers in the Secretariat of Planning, was conducted under the horizontal integration umbrella although not funded by the Project. In any event, the link is not apparent between the Horizontal Integration Component 3 and the third specific project objective, to “strengthen the cross-sectoral integration of investments in rural areas through improved participatory planning and monitoring at the local, municipal and State levels.” Moreover, the project funds projected in the PAD for Component 3 (US$0.82 million) were not spent. The ICR explains that the State Government (Secretariat of Planning) was “not committed” to the Component (pp.6, 21).

The Loan Agreement refers to a Human Development Index, and the PAD and ICR refer to such an index for Sergipe State as a whole and to a Municipal Human Development Index covering 71 municipalities in the State. This information was used to describe characteristics of Municipalities of the State and establish their eligibility to participate in the Project. But nowhere are these indices framed as objectives.

The linkage of the Results Framework Analysis to the project objectives is mixed. There are 21 so-called "Intermediate Outcome Indicators." It is difficult to see how some of them link to the 7 “Project Development Objective Indicators” (“PDO Indicators”) or to the three specific Project Development Objectives. Virtually all the so-called intermediate outcome indicators are output indicators, with the exception of Indicators 1 and 4, which could be viewed as intermediate outcomes and are stated in terms of families and communities benefited, respectively. However, no definition of "benefited" is provided. Two of the seven PDO indicators (2 and 3) are clearly outcome indicators and have implications for the project development objectives. The other five PDO indicators are at the intermediate outcome or output level. Their contribution to project objectives is conceivable, but in three cases assessment of achievement is not possible owing to measurement problems.


4. Achievement of Objectives (Efficacy) :

Efficacy is assessed by each of the three project objectives.
Objective 1: Improve the access of the rural poor to basic socio-economic infrastructure.
While efficacy at the output level had minor to moderate shortcomings, the lack of adequate data to assess outcomes results in a modest rating for Objective 1.
Outputs
  • 74% of project financing for socio-economic infrastructure subprojects went to the 41 poorest rural municipalities (target:70% against a baseline of zero)
  • 22,006 families benefited with socio-economic infrastructure investments (target: 20,000 families against a baseline that over 27,000 families benefited in Phase I project)
  • 547 community socio-economic infrastructure subprojects financed (target: 500 community subprojects implemented [revised from 1,000] against a baseline of 1,030 subprojects financed in Phase 1) Comment: "implemented" and "financed" are not necessarily the same.
  • Female-led associations with financed subprojects comprised 30-35% of all subprojects (target: 35% against a baseline of 31% in Phase I)
  • 40% of eligible quilombola (Afro-Brazilian minority) communities benefited from socioeconomic infrastructure investments (target: 50% against a baseline of 74% in Phase I project)
  • All eligible quilombola communities represented in Municipal Councils (target: 75% against zero baseline)

Outcomes
  • Household incomes of project beneficiaries changed by an unknown amount (target: increase of 30%, including incomes of "special" (minority) populations against an indeterminate baseline) Comment: it was not possible to calculate a percentage increase, owing to the lack of achievement and baseline data. This was one of two unambiguous outcome indicators among the 28 indicators in the ICR Results Framework Analysis.
  • Value of household assets of project beneficiaries changed by an unknown amount (target: increase of 20% against an indeterminate baseline) Comment: this was also an outcome indicator, but it was not possible to calculate a percentage increase, owing to the lack of achievement and baseline data.
  • Municipal Human Development Index. No information about change in this index during the project period.

Objective 2: Raise incomes and capital assets through investments in productive activities and increased linkages between small producers and markets. The ICR states that:
"this sub-objective was essentially the Project’s "core." Its success depended on the financing and management of sustainable productive activities which could be marketed in an organized manner ... under contractual arrangements." (pp.14-15) The performance of this objective at the output level was less satisfactory than for Objective 1. Taken together with the lack of information at the outcome level, efficacy of achievement of Objective 2 is rated negligible
Outputs
  • 30.1% or 74 of productive subprojects were led my women (target: 35% against a baseline of zero)
  • 69% of beneficiary associations surveyed for the Final Evaluation accessed credit from other sources after their subproject was financed (target: 30% against a baseline of zero)
  • 30% of households with productive investment subprojects had contracts with commercial buyers (target of 70% against a baseline of zero) The ICR explains that the implementing agency (Sustainable Development Company of Sergipe) "lacked technical capacity to lead an organized strategy and target was unrealistic" (p.iv)
  • About 23% of productive beneficiaries received some form of regular TA (technical assistance) focused on small-scale agriculture (target: 80% of productive investments with regular TA against zero baseline)

Outcomes
  • No results at the outcome level.

Objective 3: Strengthen the cross-sectoral integration of investments in rural areas through improved participatory planning and monitoring at the local, municipal and State levels. The ICR cites the following achievements for the third objective, but it is not clear how any of these results measure “cross-sectoral integration of investments through improved local participatory planning and monitoring.” Efficacy of achievement of this objective is rated modest.
Outputs
  • The project is claimed to have leveraged US$6.00 of additional resources for every US$1.00 of project investment funds from other programs and projects (target: US$5.00 against a baseline of zero)
  • 86% of Municipal Development Councils "were involved in leveraging activities" from Federal and State programs (target: 50% against a baseline of zero) Comment: the ICR is internally inconsistent when it claims in a comment that "all 71 Councils" or "200%" leveraged resources from other programs (p.vii).
  • An improvement of 30% in a “social capital index” was to occur by the end-of-project, but achievement was not measured since the data collected did not align with the index variables.

Outcomes
  • The third indicator above (“social capital index”) might conceivably have been an outcome. But its lack of definition makes it impossible to determine its achievement. As noted above, it was not measured.

Additional Comments
There are several other achievements cited by the ICR Framework that cut across the specific project objectives. In some cases, it is not clear whether the achievement is an outcome or an output.

Training
Four training outputs are presented. The achievements met or exceeded targets in each case (although the training of State Secretariat of Planning personnel was not financed by the Project).
  • Municipal Development Council Members
  • State Secretariat of Planning middle managers and technical personnel
  • Sustainable Development Company of Sergipe (the implementing agency) staff
  • Beneficiary Associations entering the Project

Evaluation and Related Studies The Results Framework Analysis includes information on four evaluation-related indicators. These are discussed in section 10 on M&E.

Human Development Index
The ICR provides no way of measuring change in this index, cited by the Loan Agreement as the “Borrower’s Objective.”

Other The Framework includes several other indicators dealing primarily with internal project management, namely, issuing an Operational Manual, including Safeguards and anti-corruption policies, staff of the implementing agency visits to subprojects, and establishing and upgrading a functioning MIS (Management Information System, which was a Project subcomponent but funded entirely by the Government). All these targets were achieved.

5. Efficiency:

Due to multiple factors adversely affecting project implementation, efficiency is rated modest. The cost-benefit analysis of efficiency by the ICR is limited to a random sample of 37 subprojects financed under Project Component 1 (the total universe was 794 subprojects financed by the Project, which accounted for almost 90% of total project costs). A weighted average is obtained to yield an estimated Economic Rate of Return (ERR) of 22%. This average hides the fact that the rates of return on individual subprojects are skewed. From 32%-to-46% of the analyzed subprojects yielded returns below the 10% cutoff employed by the Bank (implying a negative Net Present Value). The ex ante (before-project) ERR estimated at Project appraisal was 25%. This was based on economic analyses of previous projects in twelve categories that corresponded to those planned for the current project

However, the Project was beset with a number of implementation difficulties that adversely affected efficiency (pp.6-9). Some of these were apparent at entry and will be discussed further in section 8-a. Others relate to the Borrower and will be discussed in section 9. The ICR sums up the implementation of the project as “dysfunctional” (p.7). The following are the most salient manifestations of problematic implementation:
  • Weak management on the part of the implementing agency, lacking a consistent, efficient approach.
  • In comparison with the Phase I Project, the Phase II project experienced a marked increase in bureaucracy in Government layers, affecting all stages of subproject approval, resulting from a "host" of new legal, fiscal, environmental and documentation requirements.
  • Government-required re-registration of community organizations and restructuring of Municipal Councils.
  • A State Budget Law prohibiting funds transfer to non-governmental organizations deemed of “public utility," subsequently replaced by a 2-year waiting period before a new association could participate in the Project.
  • A supply-driven approach promoted by the State Government that led to widespread acquisition of tractors, raising concerns about land management and environmental impact. Tractor cost almost doubled and the justification for using grants for this purpose was questioned.
  • Institutional restructuring after the Midterm Review that "virtually paralyzed" project execution through to closing.
  • Reduced priority over time given by the State Government to the Project in favor of Federal cash transfer programs.
  • Disbursement was extremely slow until the final 6 months when almost half of planned disbursements occurred, which raised concerns of the Bank team about the quality of subprojects..
  • Actual cost of Component 4 (Administration) exceeded appraisal cost by 311% ($1.68 million vs. $0.54 million).

The Project was extended a relatively brief time, from 12/31/2011 to 06/30/2012, although the Borrower had originally requested an extension of 18 months to permit full disbursement of remaining loan funds and apparently expected a further extension after the first six months (ICR, p.8). But the Bank refused to consider any extension beyond six months and $2.31 million of the loan was canceled. Even with the six-month extension, the effective implementation period of the project was only three years.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
25%
86.7%
ICR estimate:
Yes
22%
87.3%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Outcome is rated Moderately Unsatisfactory.
Notwithstanding shifting priorities in approaches, the first two project objectives are still relevant, reflecting priorities of the Bank and the Government. However, the third objective suffers from ambiguity between being an objective and being a means. Moreover, it was difficult to measure achievement of this objective (suggested by the ICR, p.6). The relevance of design suffers from a large number of indicators, mostly at the output level, that do not link clearly to the project objectives. Efficacy of achievement was modest at the output level for both the first and second objectives. Efficacy could not be assessed at all at the outcome level, owing to lack of baseline data for the first objective and absence of any outcome level data for the second and third objectives. Therefore, efficacy for the first objective is rated modest, for the second objective is rated negligible and for the third objective, modest. Although the Economic Rate of Return, based on a sample of subprojects, was 22%, up to 42% of those subprojects had an ERR of less than 10%. Moreover, implementation was beset with numerous delays that resulted in half of allocated resources being disbursed in the last 6 months of the project. The Borrower had requested an extension of 18 months but in view of the problematic implementation performance, the Bank agreed to only a 6 month extension at the end of which the IBRD loan balance of US$2.31 million was canceled. Efficiency is rated modest.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

The main risks at closure relate to the second project development objective: "… raise incomes and capital assets through investments in productive activities and increased linkages between small producers and markets." These risks are rated significant-to-high. The risks at closure for the first project objective, “…improve the rural poor’s access to basic socioeconomic infrastructure,” are rated moderate. There is insufficient information to rate the risk at closure for the third objective. Overall, the risk to development outcome is rated significant.

Risks anticipated at appraisal materialized during implementation, but mitigation measures were not adequate to address them (ICR, pp.5-6,19). Some of these risks were still apparent at closure.
  • Institutional. The institutional capacity of the implementing agency, the Sustainable Development Company of Sergipe, was over-estimated for at least two reasons (i) it was moving into the new territory of productive, commercial subprojects, and (ii) the fact that skills erosion caused by the agency’s disengagement between Phase I and Phase II projects should have been perceived as a risk. The involvement of the Agricultural Development Company toward the end of the project may have at least partially mitigated these weaknesses.
  • Technical.
      • Weak capacity to design productive subprojects, including the demands of bringing them to market and undertaking analyses of local economic conditions, value chains and productive clusters.
      • Lower supply and quality of technical assistance, rural extension and training than assumed at appraisal.
      • Failure to implement an operational marketing strategy.

Other factors influencing risk at closure include:
  • Economic.
      • About half of analyzed productive subprojects had marginal or negative returns or prospects.
      • Although the situation was expected to improve by closure, only about 40% of subprojects were reported to have operations and maintenance “under control.”
  • Environmental Concerns were expressed in the end-of-project survey about environmental soundness of tractor investments and insufficient training and awareness-building in subproject environmental management.
  • Likelihood of Future Bank Support The Bank is not likely to provide future support for the ”Northeast Community-Driven Development model” in Sergipe. (ICR, p.19)

    a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:
Notwithstanding several positive factors contributing to quality at entry, there were significant shortcomings. Quality at entry is rated Moderately Unsatisfactory.
Among the positive factors planned or built into the project at entry were:

  • Use of tested methodologies and incorporation of lessons
  • Transfer of investment financing directly to communities
  • Targeting arrangements were well-defined and vulnerable groups were a priority
  • Safeguards based on consultation with ethnic/indigenous peoples.

Among the negative factors were:
  • Inadequate attention to the potential impact on capacity and costs of the hiatus between Phases I and II
  • Overly ambitious PDO, indicators and evaluation program
  • Evidence of hasty preparation of some indicators and insufficient attention to design of M&E in general
  • Inadequate assessment of implementing agency capacity to lead a small farmer marketing strategy/campaign
  • Insufficient consultation with stakeholders including the Municipal Councils, which might have detected widespread fiscal irregularity of community associations earlier as well as the erosion of their capacity.

By the time the Project became effective, institutional capacity on the ground was not yet able to fulfill the requirements of the Project strategy.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
While there were positive aspects of supervision, there were also significant shortcomings. Quality of supervision is rated Moderately Unsatisfactory.

Among the positive factors were:

  • Supervision missions were regular and reported
  • Attention was paid to Safeguards and fiduciary issues, and to following-up agreed actions
  • The Project was complex and difficult to supervise but benefited from an experienced Bank team and a single TTL throughout its execution
  • The Bank team consistently provided support and guidance to the Borrower to improve implementation.

Among the negative factors were (ICR, p.20):
  • A very slow disbursement and its underlying causes merited a stronger response from the Bank and its consequences conveyed clearly to the Borrower
  • The lagged implementation of environmental subprojects and their potentially valuable demonstration effects needed more supervision attention
  • Excessive evaluation requirements for each subproject warranted streamlining and re-design consistent with the time remaining
  • The flawed Results Framework should have been restructured early on.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
There were significant shortcomings in performance of the Government of the State of Sergipe, which leads to a Moderately Unsatisfactory rating.
The State Government took the following actions, which supported the Project:

  • The State 2008 budget allocation to the implementing agency, the Sustainable Development Company of Sergipe, was largely earmarked for the Project
  • Unification of the Municipal Councils into Municipal Sustainable Development Councils made them more effective participatory bodies
  • Institutionalizing participatory planning mechanisms originally piloted by Bank-supported CDD operations
  • Adoption of a state-wide approach to facilitate the cost-effective integration of programs
  • Proposing a pilot operation to test a methodology for scaling up, which partially influenced the design of a proposed DPL.

The following reflected lack of support (ICR, pp.6, 20-21):
  • Declining interest in the Project due to availability of alternative development resources
  • Municipal Councils lacked information and resources to promote independence and improve planning
  • “Bureaucracy and multiplication of regulations” caused an inconsistent approach to the Project’s participatory focus on community associations and Municipal Councils, leading to bypassing participatory channels in such cases as tractor procurement (ICR, p.6)
  • Erratic counterpart funding performance until the last six months of the project
  • Moving the implementing agency and replacing the project Coordinator and managers at a critical time when Mid Term Review recommendations were to be implemented in order to accelerate implementation
  • The State Secretariat of Planning did not agree to finance the “Horizontal Integration” objective/ component with project funds as agreed.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:
There were significant shortcomings in performance of the implementing agency, the Sustainable Development Company of Sergipe, which leads to a Moderately Unsatisfactory rating. (ICR pp.6-9, 12-13, 21)
The implementing agency faced many difficulties, some of which were beyond its control, yet showed good performance inter alia, in meeting or exceeding some basic targets for subprojects and beneficiaries, adhering closely to project targeting design, to the inclusion of vulnerable populations, and to leveraging complementary resources.

Nevertheless, the following factors, taken together, constitute significant shortcomings:

  • Lack of a consistent, efficient approach to project management and on occasion disregard for, or inability to generate, an organized response to Bank efforts to help improve performance
  • Extremely slow progress throughout and rushed execution in the final stages, with uneven impact on quality in the final six-month stage
  • Mismanagement of procurement in specific instances, in particular, tractors and water cisterns
  • Weak performance on some parts of the PDO, in particular, the second and third specific objectives
  • Non-disbursement of 10% of the Loan
  • Failure to conduct key studies: the baseline study, the physical performance study, and the impact evaluation and case study components of the final evaluation.

In sum, the implementing agency lacked the capacity to manage the elements, including market development, required to achieve the second project objective on income and asset generation, and for the third objective on cross-sectoral integration, to proactively leverage resources or improve participatory planning and monitoring.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
Some elements of M&E design were too demanding and in other respects they were inappropriate or incomplete. The three, somewhat disparate project objectives made for a complicated and ambitious overall design. The ICR concludes that the 3 objectives and the subsidiary indicators were too complex and demanding for the capacity of the implementing agency and the technical assistance available to help realize the objectives (p.5). There also appear according to the ICR to have been mixed signals/misunderstandings at negotiations on targets for subprojects and beneficiaries (p.6).

The Key Project Outcome Indicators assumed strong technical capacity and a monitoring and evaluation effort (including participatory monitoring) of sufficient scope and focus to capture outcomes at the local, municipal and State levels. The 21 Intermediate Outcome Indicators were too numerous, of uneven quality and lacked baselines in many cases. Some indicators were more typical of legal covenants. Others referred to internal processes rather than outputs or outcomes.

The 70% of small farmers targeted to benefit from productive investments through commercial contracts by closing was very difficult to meet, and reflected incorrect/overly optimistic assumptions about market viability and farmer capacity. The key indicator intended to measure achievement of the third project objective, by calling for the Project to “leverage” additional financing of 5:1 (leveraged funds to project resources) from other programs, could not capture the complexity of “cross-sector integration.”

In addition to the Management Information System (MIS) established under the previous Phase I project to monitor project performance indicators, review monthly disbursement summaries and supervise implementation progress, the current project also planned an ambitious slate of evaluation events: a Baseline Study, Mid-term Review Study, annual Physical Performance Studies, a Final Impact Evaluation with quasi-experimental design, case studies and a Borrower Completion Report. Design for the impact evaluation was at an advanced stage by appraisal. This seems to be an overly ambitious evaluation design for a project of this scope and time frame. (The ICR deems the design "excessive" -p.6)

b. M&E Implementation:
The Management Information System (MIS) produced a data base on the subproject cycle, physical performance and fiduciary aspects. But it did not develop the planned capacity to measure changes in and linkages between project investments nor the wellbeing/welfare, poverty reduction and/or changes in behavior of beneficiaries.. The Project financed a completed Mid-term Review study (2011), and a Final Evaluation (2012). Terms of reference were finalized late in the Project for planned Baseline and Physical Performance studies, but they were not conducted. Monitoring and evaluation activities were not designed to capture certain Key Indicators (updated Social Capital Index, value of capital and household assets, impacts of product commercialization), or some of the Intermediate Outcome Indicators. The ICR suggests that the implementing agency may not have fully understood the PDO, did not pay attention to monitoring the Results Framework over time, and focused primarily on numbers of subprojects and beneficiaries (p.10).

a. M&E Utilization:
MIS outputs and the findings of the Final Evaluation were used for the ICR. Dissemination of project results was carried out only in broad outline. The Final Evaluation was delivered after closing so any dissemination events were outside the purview of the Project. Since the implementing agency prepared videos and other materials showing certain key results, it is presumed that dissemination was planned.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
The Project triggered the following Safeguards: Environmental Assessment (OP4.01), Natural Habitats (OP4.04), Pest Management (OP4.09), Physical Cultural Resources (OP4.11) and Indigenous Peoples (OP4.10). The ICR discusses compliance with the Environmental and Indigenous Peoples Safeguards. The Project was designated Environmental Category B, indicating that its potential environmental impact on human populations and ecologically important areas was considered moderate or modest. The Project had an Environmental Management Plan (EMP, 2008) and an Indigenous and Afro-Descendent People’s Development Framework. While small-scale, all individual investments were expected to comply with Federal and State environmental laws and new screening and other requirements. Environmental screening of proposed subprojects was standard practice. The implementing agency established an Environmental Quality Technical Group (NTGA), and offered financing for new, demonstration environmental subprojects.

Environmental Safeguards compliance was satisfactory, as determined by Bank specialists during field supervision, but the demand for environmental demonstration subprojects and associated training was weak. The Technical Group Survey data show that demand-driven rural projects in the Northeast - an environmentally fragile region - would benefit from expanding their vision beyond screening and compliance to investment in conservation and recovery. Demand for environmental subprojects was modest due to the implementing agency’s lack of action in building community awareness. The ICR observes that since 57% of beneficiary respondents said there was no environmental impact from their subproject and 38% did not respond; no measures were introduced to study, minimize or resolve environmental impacts (p.11). Of the 55 conservation subprojects proposed, none was financed.

The Project’s inclusion of indigenous peoples and quilombola communities was governed by a Development Framework/Strategy prepared in consultation with such groups, public authorities and specialists and widely discussed at a series of Municipal Conferences for Territorial Planning coordinated by the State Secretariat of Planning. Consistent efforts were made by the implementing agency to ensure that these groups had access to and benefited from the Project. There were 18 subprojects financed for 10 quilombola communities (mostly cisterns, fishing equipment, cultural activities, and digital inclusion) valued at US$726,000 or an average cost of some US$40,300, on par with the overall average unit cost.

b. Fiduciary Compliance:
Financial management supervision missions by Bank specialists in February 2008 and September 2009 rated project financial arrangements as Satisfactory and FM risk as Moderate. Supervision in October 2011 downgraded the FM rating to Moderately Unsatisfactory with control risk rated Moderate due to some deficiencies likely to affect the provision of timely information needed to monitor resource management and project implementation. This rating was maintained by the final FM supervision in March 2012.

Audit performance was Moderately Satisfactory. The first audit covering project effectiveness (July 17, 2009) through end-December 2010 was received late and the auditors’ opinion was qualified. A revised audit addressed key points to the Bank’s satisfaction. It was agreed that the Borrower would present the next two audit reports on June 30, 2012 (covering January 1, 2011 to December 31, 2011) and April 30, 2013 (covering January 1, 2012 to October 31, 2012). The 2011 audit report was delivered on time and was unqualified. The 2011 audit report asked the implementing agency to conduct a technical audit of community subprojects, particularly those executed in the final semester. This exercise was to be self-funded and the report was delivered by March 2013.

An initial Procurement Post-Review in 2009 rated the project Satisfactory overall with a risk rating of Average. Follow-up during the Mid-term Review in 2010 found that community procurement complied with the Procurement Plan and Operational Manual. Due to some recommendations from the previous review not having been implemented, the risk rating for procurement processes was rated Substantial and the overall rating downgraded to Moderately Satisfactory. Procurement performance was then downgraded to Moderately Unsatisfactory by the final supervision mission due to issues affecting price quotations for water cisterns. A set of measures to increase overall transparency of procurement processes was discussed and a follow-up Independent Procurement Review analyzed 12 such subprojects resulting in an Action Plan. The Bank’s Procurement and Financial Management specialists reported that the implementing agency’s compliance with the Plan had been satisfactory. The agency adopted all of the Bank’s recommendations.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Risk to Development Outcome:
Significant
Significant
The ICR rates Risk to Development Outcome as "Substantial" (pp.i, 19). The Risk ratings range is Negligible to Low; Moderate; Significant; High. IEG rates Risk Significant. 
Bank Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The following lessons are selected and edited from the ICR (pp.21-22):
1. Fomenting community demand without commensurate capacity or resources to filter, approve and finance the resulting proposals may be counter-productive. Local governments need to have credible information on the likely financing to be available as well as the capacity to design messages tailored to targeted groups on both poverty and capacity grounds, drawing on relevant skills to ensure proper analysis and differentiation of proposals according to social or commercial objectives. In the absence of these prerequisites, communities may not be ready to go along with another government program that promises but does not deliver.
2. The sourcing and quality of technical assistance is as important as its financing and needs attention from the earliest stages of project development. Allocating a portion of project/sub-project cost for technical assistance is not sufficient to resolve the fundamental problem of the scarcity and quality of such services in rural areas or the need to build up those services over time. Studies and direct experience, such as that of the Sergipe State Integrated Rural Poverty Project, consistently show that TA quality and quantity are a top priority for rural development stakeholders in mainly rural, relatively low income jurisdictions.
3. The design of the monitoring and evaluation framework for a given project must take
into account the information expected at end-of-project and how the project will obtain it. Starting with a clear understanding of project objectives and their link to the results framework, fewer large scale studies and more issue-targeted analyses should provide evidence on PDO achievement by closing. If certain indicators are unsuitable or unrealistic, adjustments should be addressed as early as possible. Turnover in the Borrower’s management team requires an intensive effort to help the new group understand and buy into project objectives, the results framework and project methodology and to keep M&E on track.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR draws on a range of evidence, including from the predecessor Phase I project. Its analysis is on the whole sound and internally consistent. It is quite candid in its assessments of each factor, particularly of the Results Framework Analysis and Efficacy, Bank and Borrower Performance, Risk, and M&E. It is generally consistent with the ICR guidelines and its section on Safeguards and compliance is comprehensive.

The Project Team sought under challenging conditions to substantiate project achievements, but more could have been done to measure results at the outcome level in particular (see section 4 above). Without such an assessment, there is little objective, credible basis for concluding that Project Development Objectives were achieved.

The ICR is not clear with regard to the extension of the project, the reasons for it and its length (e.g. see p.8).

The ICR has a tendency for long sentences and awkward constructions that interfere with meaning – for example,

    In retrospect, the risks associated with institutional inactivity between projects were not perceived, nor was the potential issue of cost escalation, the signs of which would have been evident at appraisal and represented a risk to project scale and impact since the most common response is to reduce targets.” (p .6)

    a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions