Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Second Poverty Alleviation Fund Project


  
1. Project Data:   
ICR Review Date Posted:
12/23/2013   
Country:
Pakistan
PROJ ID:
P082977
Appraisal
Actual
Project Name:
Second Poverty Alleviation Fund Project
Project Costs(US $M)
 368  1,205
L/C Number:
C3834
Loan/Credit (US $M)
 238  568
Sector Board:
Cofinancing (US $M)
 107  637
Cofinanciers:
NA
Board Approval Date
  12/04/2003
 
 
Closing Date
07/31/2008 06/30/2011
Sector(s):
Micro- and SME finance (70%), Other social services (10%), Water supply (10%), Irrigation and drainage (5%), Roads and highways (5%)
Theme(s):
Rural services and infrastructure (23% - P) Urban services and housing for the poor (22% - P) Gender (22% - P) Improving labor markets (22% - P) Micro Small and Medium Enterprise support (11% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Roy Gilbert
Robert Mark Lacey Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The project appraisal document formulated the project aims in the form of an overall project development objective PDO (PAD p. 2) and a set of six specific objectives (PAD pp. 37-38). This formulation was retained in subsequent Project Papers for Additional Financing (AF):

Overall project development objective: To reduce the incidence of poverty in the country through provision of resources and services to the poor and low income, particularly women.
Specific objective 1. Wealth levels of the poor to increase significantly due to micro-finance.
Specific objective 2. Pakistan Poverty Alleviation Fund (PPAF) and partner organizations (POs)* moving towards financial sustainability and enhanced institutional capacity.
Specific objective 3. Increased access of poor, especially women, to micro-credit and skill development training.
Specific objective 4. Increased access of poor communities to infrastructure.
Specific objective 5. Initiate provision of social services to the poor

* 'Partner organizations', (POs) are important project stakeholders and executing agencies; they are defined in the PAD as all those non-governmental organizations (NGOs), rural support programs, community-based organizations, and private sector institutions and entities that are involved in the poverty alleviation work of the Government's Pakistan Poverty Alleviation Fund (PPAF). POs are expected to perform the following functions, depending upon arrangements reached with the PPAF:

  • Mobilizing communities and training them in skill enhancement, marketing, etc.
  • Acting as intermediaries for providing micro-credit loans
  • Providing communities with health and education facilities
  • Acting as intermediaries for small-scale infrastructure projects in supervising and monitoring PPAF-financed projects; assisting communities with the preparation of their proposals; and facilitating communities in implementation of projects (PAD pp. 19-20)

In the development credit agreement (DCA) the project objective has similar content to the PDA version of the overall PDO although without explicit reference to women and without the five specific objectives. In the words of the DCA, it is as follows: "The objective of the Project is to alleviate poverty by improving the access of the rural and urban poor to economic resources and services." (DCA Schedule 2 p. 19)

This Review uses the PAD formulation as it is more monitorable.

Through three amendments invoking Additional Financing (AF) for the operation, the project objectives remained unchanged.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:
As noted in the PAD and additional financing (AF) documentation:

1. Micro-Credit and Enterprise Development Loans (original appraisal cost - US$258 million; estimated AF cost - US$500 million; actual cost - US$764 million). The provision of one million new loans for over 0.5 million borrowers by disbursing US$235 million for micro-credit and an additional US$23 million to cover the delivery costs incurred by POs.

2. Small Scale Infrastructure Projects (original appraisal cost - US$71 million; estimated AF cost - US$20 million; actual cost - US$112 million). Provision of grants on a cost sharing basis to finance small scale infrastructure projects identified by community organizations.

3. Education and Health Projects (original appraisal cost US$5 million; estimated AF cost - US$14 million; actual cost - US$22 million). Provision of grant money to make quality education and health services available to the poor, especially women.

4. Training and Skill Development (original appraisal cost - US$9 million; estimated AF cost - US$5 million; actual cost - US$11 million). Provision of grants for operational support and the training of communities and staff of POs and PPAF.

5. Capital and Operating Costs of PPAF (original appraisal cost - US$15 million; estimated AF cost - US$6 million; actual cost - US$14 million). Reimbursement of incremental costs for PPAF arising directly from the project, including: (i) construction of new PPAF office in Islamabad, to allow PPAF to move from rented premises; (ii) equipment for the new building; (iii) consultancies primarily for socioeconomic baseline collection; (iii) support to incremental PPAF operating costs on a declining basis

6. Equity for PPAF (original appraisal cost - US$10 million; estimated AF cost - US$0 million; actual cost - US$10 million). As per the PPAF-2 DCA, GOP provided a second injection of equity into the endowment intended ensure the feasibility of a larger scale operations by PPAF and their sustainability into the future.

The six project components above were not revised; however, three subsequent components were added to in response to the earthquake in Oct. 2005.

7. Disaster Recovery (original appraisal cost - US$0 million; estimated AF cost - US$5 million; ; actual cost US$5 million). In order to support the relief efforts in the areas of Khyber Pakhtunkhwa, and Azad Jammu/Kashmir provinces that were badly hit by the 2005 earthquake. For this, IDA agreed to divert US$5 million from Component 2 (Small Scale Infrastructure Projects).

8. Emergency Relief, Rehabilitation and Reconstruction - (original appraisal cost - US$0 million; estimated AF cost - US$238 million; actual cost - US$242 million ). Responding to the 2005 earthquake, a component was added through two amendments to the original DCA (Dec 2005 and May 2007) to address the needs of restoration, rehabilitation and reconstruction of earthquake affected villages and communities.

9. Participatory Development through Social Mobilization (original appraisal cost - US$0 million; estimated AF cost - US$75 million; actual cost - US$25 million). Mobilization of groups formed around micro-credit and infrastructure support where insufficient attention had been paid building institutions to empower the poor, give them voice to raise their concerns and issues, and to use collective resources to enhance their incomes and productivity.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
The original (October 2003) appraisal estimate of total project costs of US$368 million, rose to an actual total project cost at completion (June 2011) of US$1,205 million. According to the ICR, nearly all of the increased cost was funded from two sources: (i) US$493 million from greater sub-borrower microcredit repayments than were expected at appraisal (partly due to three year extension of project closing); and (ii) US$313 million of Additional Financing (AF) by IDA. AF for this project was approved three times: in December 2005 for US$100 million, in May 2007 for US$138 million, and in June 2009 for US$75 million. Consolidating appraisal and AF commitments brought the project's total estimated cost to US$1,181 million. The project's original closing date in July 2008 indicated an initially intended implementation period of about 4.5 years. Amendment agreements that included AF extended the closing date to June 2010. This was postponed one more time to June 2011, extending the actual implementation period of the project to 7.5 years.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Substantial
The overall project objective of reducing poverty remains consistent with both Bank and country priorities. The 2010-2013 Country Partnership Strategy for Pakistan provides explicit support for a series of Poverty Reduction Support Credits (PRSC) (CPS 2011 p. 6). Poverty reduction also features prominently in several of the CPS's intended outcomes (CPS 2011 pp. 28, 29). Moreover, the project objective's focus on women is entirely consistent with the CPS's continuing attention to improving female access to credit (CPS 2011 p. 23). The objective is also consistent with the Millennium Development Goal (MDG) of reducing the share of all people living in extreme poverty. On the relevance of the project objective to Pakistan's own development priorities, however, the ICR is silent. IEG notes that Part 2 of the Pakistan's six-part Medium Term Development Framework 2005-2010 (the most recent edition found) is focused entirely upon achieving the MDGs, including poverty reduction and promoting gender equality to which this project was committed (http://www.pc.gov.pk/mtdf.html). In its principal manifesto, the new government of Pakistan has explicitly highlighted the role of micro-credit as an instrument of choice for poverty reduction, citing the PPAF by name (PML National Agenda for Real Change 2013 p. 14).

b. Relevance of Design:
Modest
The PAD's Project Design Summary (PAD p. 37-39) includes the elements of a results chain linking project actions to the achievement of objectives and beyond, toward contributions to the MDGs. The linkages could be more explicit, however. Some of them, such as those 'from outputs to outcomes', such as continuing good governance of the program and high repayment rates do not address the objective's poverty reduction or gender equality objectives explicitly. Instead they seem to be more directly concerned with sustaining the project activities themselves. These linkages would have been clearer if the project design had had clear and quantified outcome targets. The ICR itself notes that there were discrepancies between the targets in the PAD text and those in the Project Design Summary (ICR p. 8).


4. Achievement of Objectives (Efficacy) :

Overall objective - To reduce the incidence of poverty in the country through provision of resources and services to the poor and low income, particularly women: rated Substantial
The PAD reported that the head count ratio showed that 30% of Pakistan's population lived in poverty in FY2002-2003 (PAD p. 3). The ICR itself does not update the head count ratio or provide another measure of poverty that could clearly demonstrate poverty alleviation or reduction in Pakistan by project completion as intended. The Bank's public webpage for Pakistan [http://data.worldbank.org/country/pakistan] gives head count data for 2006 as the most recent year when it had fallen to 22.3%. While this indicates a significant initial improvement from the baseline, but it still leaves unknown the trend, up or down, over the past six years. Indirectly confirming the lack of data, a recent Ministry of Finance (MOF) report announced that a committee of poverty experts had been set up to estimate Pakistan's poverty head count (MOF Pakistan Economic Survey 2010-12 p. 223) . Up to date data on poverty alleviation or reduction is therefore still needed to make the case that this project did indeed achieved its principal objective at completion. Responding to this need, the Region later guided IEG to more recent poverty data for Pakistan. A poverty headcount rate of 12.4% for 2010/11 was cited in the Bank's SASEP October 2012 Pakistan Economic Update (p. 19) by Saadia Refaqat and M. Waheed, that called Pakistan's declining poverty trends "a puzzle" (p. 19), the report's diffidence expressed by the caveat "if poverty has decreased" that introduces its conclusion on this topic (p. 20). The Economic Update acknowledges the media as being the source of this data, namely The News of September 6, 2012: http://www.thenews.com.pk/Todays-News-3-130225-Govt-to-release-data-of-PSLM-Survey-2010-11. The newspaper reports the government as the source of this data, while at the same time noting the political controversies surrounding the irregular availability of time series of Pakistan's poverty head count data in recent years. The Economic Update, for instance, notes the Bank's own estimate of a poverty headcount of 17.6% for 2008/09 that was not released as an official figure by the government of Pakistan. While the recently reported 2010/11 poverty data still awaits ratification by the Bank and other interested parties, this ICR Review takes it to be evidence that this project substantially achieved its objective of reducing the incidence of poverty. Verification of this data would be part of a future PPAR by IEG of this project.
Specific objective 1. Wealth levels of the poor to increase significantly due to micro-finance: rated Substantial
The findings of a survey of project beneficiaries by Gallup Pakistan in 2009 indicated that the share of borrowers experiencing positive change in their personal and household income was significantly higher than that of non-borrowers. Altogether, 82% of borrowers reported achieving returns on (micro-financed) investment averaging 78% in 2009, compared with just 30% in 2002.

Specific objective 2. Pakistan Poverty Alleviation Fund (PPAF) and partner organizations (POs) moving towards financial sustainability and enhanced institutional capacity: rated Modest
The main evidence given by the ICR (p. v) in support of the achievement of this objective is the 100% repayment rate by borrowers. While guaranteed revenue like this may be necessary for financial sustainability, it is not, by itself, a sufficient condition. For that, data on the PPAF expenditure side is necessary too to demonstrate that revenues (including these repayments) were sufficient to cover the expenditures made. Data on this would come from audited financial reports and statements, evidence from which is not provided by the ICR.

Specific objective 3. Increased access of poor, especially women, to micro-credit and skill development training: rated Substantial.
For this, the ICR reports 55% percent enrollment rates for girls in 962 schools and 56% percent female visits to health facilities (out of a total of 4.6 million). Where these data fall short of what is needed to solidly substantiate the achievement of this objective is the lack of a baseline. While the female participation rates reported look good, we do not know in the absence of a baseline if they are better than they were before the project. Another indicator does indeed show an improvement, however. The ICR's reported growth in the volume of micro-credit of very small loans points to increased access by the poor, with the women's share of the total rising from 36% in 2005 to 66% in 2011 (ICR p. 24). On the other hand, information recently received from PPAF via the project team reported that the average loan size had increased by 196% over the 2005-2010 period

Specific objective 4. Increased access of poor communities to infrastructure: rated Modest.
The ICR did not provide information regarding access by poor communities to infrastructure to substantiate its conclusion that this objective was fully achieved.

Specific objective 5. Initiate provision of social services to the poor: rated Substantial
The provision of social services to the poor, and on a large scale, is substantiated by evidence such as the 4.6 million poor beneficiary visits to health facilities. Whether this constituted an initiation of this service provision or not is not well substantiated in the ICR for the lack of baseline evidence. Without it we cannot know whether even more poor people visited these facilities prior to the project. In later comments on this, the Region informed IEG that the following data from the Pakistan Social and Living Standard Measurement Survey, with baseline references showed the following results:
  • Based on mother’s recall, proportion of less than 1 year children immunized against measles increased from 76% in 2005-06 to 81% in 2011-12.
  • There was a decrease in the proportion of children under five suffering from diarrhea i.e. from 12% in 2005-06 to 8% in 2011-12.
  • About 68% of mothers in 2011-12 had at least one antenatal care consultation as compared to 52% in 2005-06.
  • The IMR declined from 69 per thousand live births in 2007-08 to 63 per thousand live births in 2011-12.
According to PPAF's own website (http://www.ppaf.org.pk/) better access to education through increasing geographical proximity, enhancing affordability, and ensuring quality of service has led to increased enrolment and retention rates of students especially girls and better leaning outcomes manifested in stronger literacy and numeracy skills.

5. Efficiency:

Substantial
Although project documentation at appraisal and completion makes frequent references to high 'returns on investment' enjoyed by beneficiaries taking micro-credit, the ICR does not report economic or financial rates of returns of these project investments. For a sample of project community infrastructure investments, the ICR reports weighted averages overall of 23.8% of ERRs and 24.3% of FRRs. The PAD referred to a target population of 5 million people (PAD p. 19) of what would have been a US$368 million project, costing US$73.6 per beneficiary. Figures recently tabulated by the Region below point to a final tally of 22.9 million beneficiaries of an operation whose final cost was US$1,205 million, or US$55.2 per beneficiary. This points to an increase of efficiency by completion.

Operational efficiency, measured by the rate of disbursement of Bank financing, also showed some improvement during implementation. Originally at appraisal, the expectation was to disburse US$238 million during a 4.5 year period. US$330 million of AF was disbursed over the next 3 years, a significant acceleration per year.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
Yes
23.8%
9%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Relevance of objectives is substantial and that of design modest. Efficacy of the overall objective is rated substantial on the basis of press reports of recently released data. Of the efficacy of the five specific objectives, three are rated substantial and two modest. Efficiency is rated substantial. Overall, outcome is rated moderately satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The risk is elevated by the continuing vulnerability of the project to natural disasters, to the uncertain and deteriorating security situation, and ongoing political instability. Nevertheless, these are likely to be similar to conditions toward which the project demonstrated resilience in the past. An important factor lowering the risk is the cash flow from micro-loan repayments, strong in the past and likely to continue into the future. On balance, these factors call for the moderate rating given here.

a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:
The quality of entry of the project was undermined by a lack of clarity of the results framework. While the unclear performance indicators, some without targets, and the uncertain links between project actions and intended outcomes primarily concern M&E, these weaknesses reflect project uncertainties at the outset; uncertainty about what the project intended to achieved and uncertainty about how it would achieve its results. Implementation was wisely based upon the well-tried arrangements of the previous operation. Nevertheless, a stronger project design could have been expected from a follow-on project such as this one.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
ICR reporting of Bank supervision is parsimonious. Supervision missions were fielded regularly, with greater frequency following the 2005 earthquake. At times, very large supervision missions with 15 or more members did not operate efficiently, according to the ICR (ICR p. 16). On the other hand, IEG later learned from the project team that supervision missions often went deep into the field, 'over-nighting' in beneficiary villages to observe sub-project results directly. Supervision missions also helped introduce more measurable performance indicators, making up for some of the weaknesses of project design. The ICR states that safeguards policies were complied with, and reports no significant fiduciary issues. Later comments by the Region added the following points about quality of supervision: (i) Project supervision was carried out by the country office, making supervision an ongoing process; (ii) in the 12 month following the 2005 earthquake, the Bank fielded four supervision missions; (iii) the missions were well managed and incorporated the necessary technical specialists.

Quality of Supervision Rating: Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
The government's role in this project was limited to transferring funds to the project executing agency PPAF. The Government's own contribution of US$10 million met its appraisal obligations but represented only a very small proportion of the resources mobilized for what became a U$1,205 million operation. The ICR reports that the government performed these limited functions in a satisfactory way, although it took more than 13 months to open a special account for the project at the outset.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:
According to the ICR, the performance of PPAF is satisfactory. The ICR reports that, expressed as percentages of the total costs , PPAF's overhead of 1- 3.5% were 'extremely low'. In absolute terms, however, PPAF's operating costs were between US$12-42 million, because of the scale of this US$1.2 billion program. The ICR also noted that at project start up, PPAF recruitment and training were slow to take off, as were initial disbursements for social mobilization.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The ICR noted that several performance indicators were unclear at appraisal and, without explicit targets, a number of them contribute little to a better understanding of the project's achievements (ICR p. 8). Several of the outcome indicators listed in the PAD in fact measure delivery of outputs as the ICR notes (ICR p. 9) , and even then are difficult to quantify. The M&E design would have been more workable had it spelled out how data collection and analysis should have been carried out. A weak M&E design in this case was the result of an ambiguously conceived results chain (in the project design noted in Section 3 above) that confused the aim of achieving the objectives with that of simply keeping the program running on a day-to-day basis.

b. M&E Implementation:
Data collection 'generated a massive volume of information' (ICR p. 10) without a systematic approach to ensure its meaningful tabulation and analysis. The 2005 earthquake led to further disruptions. Progress was, nevertheless, made in M&E during implementation. The outcome indicators were better spelled out after restructuring and they are more measurable. A web-based reporting system was introduced which contributed to strengthening M&E. Other tools, including household questionnaires, baseline surveys and user surveys, were employed to try to strengthen the monitoring and evaluation of project outcomes.

a. M&E Utilization:
Data collected through M&E was used for project resource allocation. The ICR reports that it also helped to prevent fraud after the earthquake through helping to identify and reject over 20,000 bogus compensation claims, which saved over $40 million (ICR p. 10).

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:
The project was classified as Environmental Category 'B.' Environmental Assessment (OP 4.01) was the only safeguards policy triggered (PAD p. 30). According to the ICR (p. 10), OP 4.01 was fully complied with. However, the ICR provides practically no details on safeguard compliance..

b. Fiduciary Compliance:
noncompliance.The ICR reports no cases of noncompliance. Legal agreement covenants required that no more than 25% of outstanding loan amounts and 20% of community physical infrastructure expenditures should be received by any one single. The National Rural Support Programme, the largest organization participating in the project, breached this limit, although only on one occasion according to the ICR. All audits reports of the project were submitted on time and none contained qualified findings.

There is almost no discussion in the ICR of project procurement.

c. Unintended Impacts (positive or negative):
none reported.

d. Other:
none



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
According to data recently made available, the project achieved its principal objective with only moderate shortcomings. 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Satisfactory
Moderately Satisfactory
The results framework was not clearly articulated in the project design, leading to ambiguity about what the project results should be: achievement of objectives or high scores for performance indicators focused incorrectly upon the delivery of outputs instead of the achievement of the intended outcomes.  
Borrower Performance:
Satisfactory
Moderately Satisfactory
Delays in project startup and in recruitment of key staff. 
Quality of ICR:
 
Unsatisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
The ICR provides a list of 10 project-specific and general application lessons of which the following are of particular note:
  • Community participation through cash or labor contributions, where the community manages its own finances, and the procurement of materials and equipment helped ensure strong community ownership of small-scale infrastructure and development projects. This way of managing a complete project cycle approach gives voice to the poor, enhances community members' self-confidence and demonstrates that the poor can, through self-help, improve the living conditions of their communities.
  • Despite the satisfactory micro-credit loan recovery rate reported by PPAF, significant risks still remain for partner organization lending to communities. Drought losses, in particular, have given rise in the past to significant repayment difficulties faced by PO beneficiaries in a few provinces. The introduction of disaster insurance to provide coverage for borrowers who have lost everything (not included in this project) could mitigate the risk of non-repayment in areas prone to natural disasters, .

IEG adds the following lesson:
  • Especially when an ICR acknowledges shortcomings in the M&E system, as it did in this case, its self-evaluation of project performance should not be based exclusively, as it was here, on M&E performance indicators some of which it recognized as being weak. In such cases it is especially important for the evaluation to look beyond them, ideally by presenting additional evidence, or at least indicating where that evidence should be sought if it is not already available.

14. Assessment Recommended?

Yes
Why?
To include the verification of the most recent poverty and beneficiary data that supported this ICR Review's current ratings.

15. Comments on Quality of ICR:

This ICR provides good coverage of information relating to project performance indicators, but is short of a critique about the limitations of these indicators and, consequently of the data itself. These often lack baselines and measure output delivery instead of outcome achievements. This approach leaves some important gaps in the ICR assessment, perhaps the most notable of which relates to the operation's principal objective, namely whether poverty did indeed fall as a result a the project. The policy and strategic context of the project is not fully covered, partly because the ICR does not address and assess project relevance (to Bank and Government priorities) as indicated by the guidelines. Discussion of key aspects of performance, notably efficacy, Bank and Borrower performance and safeguards, is summary. The lack of supporting evidence or data leads to a key yet unsubstantiated finding such as "the bulk of PDO has been fully achieved" (ICR p. 12). Reporting project costs and financing could be more informative, especially when dealing with 'reflows' the largest single source of project funding. The report's content would also be more accessible to readers with the more careful and economic use of fewer acronyms. In some places (ICR p. 20's use of 'SMC' And "social mobilization component, for instance) the text reverts to a name fully spelled out after having previously used an acronym, leaving a reader uncertain as to whether this is a reference to a new institution altogether. The ICR's discussion of safeguards is sparse, and there is almost no information provided on project procurement.

a. Quality of ICR Rating: Unsatisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions