Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Rural Development Project


  
1. Project Data:   
ICR Review Date Posted:
06/27/2013   
Country:
Georgia
PROJ ID:
P078544
Appraisal
Actual
Project Name:
Rural Development Project
Project Costs(US $M)
 34.71  31.27
L/C Number:
C4062
Loan/Credit (US $M)
 10  10.28
Sector Board:
Agriculture and Rural Development
Cofinancing (US $M)
 14.5  10.37
Cofinanciers:
Japan: Ministry of Finance PHRD Grants, International Fund for Agricultural Development (IFAD)
Board Approval Date
  05/17/2005
 
 
Closing Date
06/30/2010 06/30/2011
Sector(s):
Crops (50%), Agro-industry (35%), Central government administration (12%), Agricultural extension and research (3%)
Theme(s):
Rural markets (29% - P) Export development and competitiveness (29% - P) Rural policies and institutions (14% - S) Regional integration (14% - S) Law reform (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Hassan Wally
Robert Mark Lacey Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
According to the Project Appraisal Document (PAD, p. 3), the Project Development Objectives were:


    "to develop the productivity and profitability of the private agriculture sector. This would be achieved by facilitating the access of Georgia’s mainly small and medium-scale farmers to supply chains, improving the competitiveness of the supply chains and strengthening the capacity of selected agricultural and financial institutions to serve private-sector agricultural market activity. This would increase incomes and employment and reduce poverty in rural areas."

According to the Development Credit Agreement (p. 14), the Project Development Objectives were:
    "to develop the productivity and profitability of the private agriculture sector by facilitating the access of mainly small and medium-scale farmers to supply chains, improving the competitiveness of the supply chains and strengthening the capacity of selected agricultural and financial institutions serving private-sector agricultural market activity."

On July 9 2009, the Board approved a level 1 restructuring, as part of which the project development objectives were changed. According to the Project Restructuring Paper, the revised objective was: “to improve agricultural production and access to markets for Georgia’s small and medium-scale farmers and rural enterprises supported by the project through: (i) increasing the competitiveness of selected supply chains; (ii) strengthening the delivery of rural financial services and of the financial intermediaries; and (iii) modernizing key institutions for food safety and property registration with direct impact for increasing competitiveness of Georgia’s agriculture.” The amount disbursed at restructuring was US$5.95 million or 58% of the fully disbursed IDA Credit.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 07/09/2009

c. Components:
1. Agricultural Supply Chain Development (Appraisal Cost US$4.27 million, actual cost US$1.08 million). To support the efficient development of marketing supply chains for commodities that have a demonstrated market potential, with the view to expand profitable domestic and export market opportunities. The proposed project would work with all agents in potentially profitable agricultural supply chains to develop and implement a holistic strategy for identifying and addressing weaknesses and bottlenecks. The project would support the following:
(a) Supply Chain Analysis and Development. To develop a holistic strategy for the expansion of profitable sales in domestic and export markets. This would involve assistance in determining consumer demand, identifying technical, regulatory, institutional, contractual and financial constraints, developing a collaborative strategy for their redress, and analyzing sources of supply. The supply chain analysis would attempt to determine where commodity associations and other market participants could productively reinforce linkages among actors along a commodity chain both formally and informally.
(b) Linkages to Farm Communities. To pilot test a program to assist farmers and communities to engage with commodity supply chains in an equitable and profitable manner. Using field demonstrations, capacity building workshops and local study tours, the project would introduce farmers to more productive and profitable market-linked agricultural technology. Leadership training would also be emphasized, especially for younger members of rural communities. Farmer linkages to agribusinesses and markets would be
strengthened and farmers would be empowered to develop marketing groups and associations with a view
to increasing product quantity and quality and, thereby, improving market access and price. The project would also provide assistance both to the new and to existing farmer groups to link to commodity supply chains. For this, the project would finance technical assistance and training and partially support farmer group set-up costs and some equipment.
(c) Technology Transfer. The project would support small-scale farmers and farmer groups engaged in potentially profitable agricultural supply chains to develop appropriate, modern farm technology, crop and livestock management practices, and post-harvest technology and demonstration programs. Development of technology would be supported through the Agricultural Supply Chain Development Fund and be coordinated by NGO and a supply chain advisory Committee.
2. Rural Finance Services (Appraisal Cost US$25.76 million, actual cost US$25.45 million). To improve the capacity of participating financial institutions to lend to the farmers, processors and agri-business enterprises involved in the marketing supply chains of marketed agricultural commodities. It included two sub-components:
(a) Credit Lines for Commercial Banks. Credit lines would be made to eligible commercial banks to increase their capacity to make medium and long-term investment loans to eligible farmers, processors and agribusiness enterprises. Loans to participating commercial banks would be either in US dollars or Euro or Lari, with a grace period of four years and repayment period of up to 10 years.
(b) Credit Lines for Non-Bank Financial Institutions. Credit lines would be made to eligible non-bank financial institutions (including Credit Unions) to increase their capacity to make small investment and working capital loans to eligible farmers, processors and agribusiness enterprises. Loans to such financial institutions would be either in US dollars or Euro or Lari, with a grace period of two years and repayment period of up to 10 years. Participating financial institutions would revolve these loans during the project. Interest rates charged by Government to participating financial institutions on dollar and Euro loans would be determined on the basis of the corresponding 6 month LIBOR (US$ or Euro) plus a spread of 2%.
(c) Strengthening the Capacity of PFIs for Sustainable Rural Lending. To strengthen the capacity of participating rural financial institutions to appraise and manage loans for production, agro-processing and agribusiness; develop and promote appropriate loan products and collateral instruments; provide matching grants to selected non-bank financial institutions for the establishment of new rural branches; and develop sustainable rural credit unions.
3. Institutional Modernization (Appraisal cost US$3.17 million, actual cost US$2.92 million). This component would focus on specific, key legal and institutional reforms that impact directly on the competitiveness of Georgian agriculture and the safety and marketability of its products and enable Georgia to meet its international sanitary and phytosanitary obligations. Interventions would be focused on:
(a) Institutional and Legal Framework. The project would work with relevant MOA staff to revise selected food, plant variety protection and veterinary laws and regulations to meet international trade and treaty requirements and support Georgian membership and participation in international standards organizations relevant to Georgian agricultural trade. Training and study tours would be provided, and operating manuals based on an EU compliant legislative framework would be developed. The project would also enable the Georgian government to establish a comprehensive, unitary food safety management and risk assessment system in line with the requirements of a draft Food Law. This would include the establishment of a Veterinary and Food Department within the Ministry of Agriculture, the strengthening of domestic and border sanitary inspection capacity, the strengthening of laboratories for accredited food inspection, the development of a risk assessment capability and the establishment of the Food Safety Council proscribed under the draft law.
(b) Support for Selected Commodity-Specific Programs. Programs to support specific commodity chains supported by project under component one and consistent with strategies and actions plans for institutional development developed under sub-component 3 (b) would be identified. Depending on the needs and proposals from the commodity chains, the project would support the development of product certification, quality testing and labeling standards. In those cases where private services are not available and public services are not proficient, the project would consider strengthening these services.
(c) Continuing Support for Property Registration. To ensure a smooth transition of Land registration and cadastre responsibilities the project would, over a two year period, complete the network of National Agency for Public Registry (NAPR) regional centers, develop systems for the integration of land and moveable property registry and land cadastre databases, establish a secure NAPR data management and transfer network and develop information programs on NAPR services. Project inputs would include training, equipment, information technology and national and international technical assistance.
4. Project Management (Appraisal Cost US$1.51 million, actual cost US$1.83 million). To finance technical assistance, training and study tour, office equipment and vehicles, staff salaries, auditing and other operating expenses related to the project.
Revised components:
In addition to the Level 1 restructuring of July 2009, there were two Level 2 restructurings (one on March 29, 2011 and the other on April 28, 2011), both involving reallocation of funds among sub-components. The main changes and reallocations resulting from all three restructurings were:
1. Agricultural Supply Chain Development. Sub-component 1 (b) – Linkages to Farm Communities, was replaced with a new sub-component 1 (b) – Training and Demonstration program that for farmers and rural enterprises involved in citrus and hazelnut supply chains.
Sub-component 1 (c) Technology Transfer was renamed the Competitive Grant Program. The sub-component was also revised in substance to allow for the possibility of extending small competitive grants to farmer groups and rural enterprises for competitiveness enhancing sub-projects in priority supply chains. The competitive grant scheme would be managed by the PCC/PIU, and the establishment of the initially envisaged Agricultural Supply Chain Development Fund was no longer required.
2. Rural Financial Services. The main changes were dropping the support for strengthening the capacity of credit unions due to the collapse of the credit union system in Georgia; and the amount of credit lines for participating commercial Banks was increased.
3. Institutional Modernization. The main change was refocusing the support for the food safety agenda by preparing the groundwork for when the legislation would be changed towards enforcing food safety actions and controls. The support included: (i) the rehabilitation of, and provision of equipment to a food safety laboratory; (ii) the rehabilitation of, and provision of equipment to several regional veterinary offices; and (iii) training and technical assistance for staff involved in the food safety agenda.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost. The total project cost at appraisal was estimated to be US$34.71 million. According to the ICR (Annex 1) the actual project cost was US$31.27 million.
Financing. The project received US$10.27 million (IDA Credit) compared to an appraisal estimate of US$10.0 million; the difference was due to exchange rate fluctuations. The Credit was fully disbursed. Cofinancing was provided by IFAD (US$8.75 million compared to an appraisal estimate of US$10.0 million) and by PHRD grants from the Japanese Ministry of Finance (US$1.62 compared to an appraisal estimate of US$4.5 million). IFAD financing was extended beyond the closing date of IDA financing to June 30, 2012, and the remaining IFAD funds were allocated mainly to Micro Finance Institution credit (ICR, p. 10).
Borrower Contribution. At appraisal, the project was estimated to receive US$2.9 million and US$4.84 million from local sources of the borrowing country and local farmer organizations, respectively, however, the project did not receive any funds from farmer organizations and received US$9.57 from local sources of the borrowing country. The Borrower contributed US$1.05 million of counterpart funds compared to an appraisal estimate of US$2.47 million.
Dates. The closing date was extended by 12 months to June 30, 2011 at the time of the Level 1 restructuring in July, 2009, to permit the completion of project-supported activities.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Original Objectives: substantial.
In 2003, agriculture contributed 18% of Georgia’s GDP and 56% of employment. The size of the agriculture sector and its role in employment highlights its importance to Georgia’s overall economic growth and prosperity (PAD, pp. 1&18). At appraisal, the objectives were in line with the Bank's December 2003 Country Assistance Strategy (CAS) for Georgia (Fiscal Year 2004-2006) which emphasized the need for further support to the rural sector, and described strategies to promote export markets and an environment for private sector-led growth. The project was also included in the priority assistance program in the Reform Support Credit presented to the World Bank Board of Executive Directors on June 24, 2004 (PAD, p. 2). At project completion, the original objectives remain relevant to the Bank's Fiscal Year 2010-2013 Country Partnership Strategy (CPS) which among other things calls for “restoring growth and competitiveness.” They were also consistent with the Government's priorities which include among other things: growth in agricultural exports, increased rural productivity and incomes, job creation, food safety and effectiveness of governance, financial intermediation and strengthening of property rights (CPS, p. 10).

Revised objectives: substantial.
The revised objectives were in line with the 2004-2006 CAS as well as the Government’s 2008 medium-term program entitled “United Georgia without Poverty”. They were also relevant to Georgia's development priorities and are consistent with the Bank's Fiscal Year 2006-2009 Country Partnership Strategy (CPS) Progress Report which called for “generating jobs” and “strengthening of public sector management”. At project completion, the revised objectives remain relevant to the Bank's Fiscal Year 2010-2013 CPS which among other things calls for “restoring growth and competitiveness.” They were also consistent with the Government's priorities which include among other things: growth in agricultural exports, increased rural productivity and incomes, job creation, food safety and effectiveness of governance, financial intermediation and strengthening of property rights (CPS, p. 10).

b. Relevance of Design:
Original design: modest.
Original design included a broad and overly ambitious project development objective. To achieve the objectives, design featured three main components that aimed to facilitate access of Georgia’s mainly small and medium-scale farmers to supply chains, improve the competitiveness of the supply chains and strengthen the capacity of selected agricultural and financial institutions to serve private sector-agricultural market activity. Most of the activities financed could be clearly linked to the goal of increasing profitability. However, the causal chain between project inputs and the other objectives was less clear. The only input that would help to raise productivity is increased access to credit. The PAD (p. 19) expressed awareness that this alone is insufficient and highlights other factors that would be needed to raise productivity, including enhanced use of machinery and mechanical services and of agro-chemicals, access to irrigation water, better terms of agricultural trade and adequate upstream and downstream technical and commercial farm services. Yet these do not feature in the project’s activities, and there appears to be nothing in the project to support them. It was unclear how component 1 would contribute to the development of supply chains and equally unclear how the support would be expected to lead to the development of specific regulatory and institutional policies. Sub-components 1 (b) and 1 (c) lacked implementation details.

Revised design: substantial
The revised statement of development objectives was clear and precise; it described target groups and provided a logical explanation of the project's intended goals. The revised results framework also included more specific, measurable inputs and expected outputs, with clearer attribution. The revised components reflected a better focus on improving productivity through the provision of technical expertise to farmers and rural enterprises combined with the strengthening of rural credit.


4. Achievement of Objectives (Efficacy) :

Original objectives:
(i) to develop the productivity of the private agricultural sector: Modest.
Outputs:
  • The project supported three supply chains as against a target at appraisal of 14. The ICR (p. 23) highlighted that this indicator was revised (target reduced) following the restructuring since its consideration against the original development objective was "not feasible." Only one grant was provided to citrus exporters aimed at achieving access to new markets through technical assistance and demonstrations for improved agricultural practices.
  • The project supported institutional development in three areas: food safety, seed legislation and cadastre compared to an appraisal target of 4: planned support to the sanitary and phytosanitary area was not implemented.
  • The project increased agricultural lending by 0.8% of agricultural GDP which represented 29% of the PAD target of 4%.
Outcomes:
The only area of support for higher productivity was through agricultural lending which experienced a considerable increase over the life of the project. From an assumed baseline of zero in 2005, 10,000 micro-finance institution credits and 27 bank loans at an average of about US$1,100 were provided. However, the ICR provides no information on the impact of project activities on increasing agricultural productivity of the private sector.
(ii) to develop the profitability of the private agricultural sector: Modest.
Outputs/intermediate outcomes:
  • The outputs relating to this objective are the same as the first two outputs relating to objective (i).
Outcomes:
The ICR (p. 23) pointed out that before the project’s Level 1 Restructuring, there was no quantitative information available on the progress of the outcome indicators of increasing agricultural production, enterprise profits and the net income of participating farmers. Subsequently, a survey of grant recipients among the micro-finance institutions and enterprises supported through the credit line to banks showed that, as a result of project activities, farmers' incomes increased by 28.3% compared to the target of 10%. However, it is not specified whether this was gross or net income. No further information on profitability is available.

Revised Objectives:

(i) to improve agricultural production for Georgia’s small and medium-scale farmers and rural enterprises supported by the project: Modest.
Outputs:
  • The Project supported 3 supply chains (citrus, hazelnut and wine) which represent 60% of the revised target of 5. Support included applied technical assistance and studies; a draft wine sector strategy; marketing and awareness-raising events; applied testing for improved production and competitiveness; and applied technical guidelines for cultivation. However, supporting supply chains lacked a consistent strategic vision. Support for the Ministry of Agriculture did not take place, and most of the PHRD funds which were to finance this activity were left undisbursed.
  • The project supported 43 farmers (direct beneficiaries) in the Adjara region and Zugdidi district through training and demonstration of new technologies for citrus and hazelnut cultivation. Demonstrations focused on the introduction of new varieties, agronomic improvements and plantation maintenance. This exceeded the revised target of 35 farmers. According to the ICR (p. 35) such new technologies contributed to increments in productivity of both hazelnut and citrus, enhanced the quality and consistency of harvest and led to better supply chain integration. The ICR (p, 25) pointed out that there were 604 additional indirect beneficiaries.
  • The project supported one rural enterprise through grants out of a target of 7.
  • The project achieved a total of US$22 million in rural investments supported by the banks which reached the target.
  • The project created 205 jobs through Micro Finance Institution lending with project funds compared to an end target of 50 jobs. While numerically the target was overachieved, the ICR (p. 25) highlighted that the initial target was very low and the indicator did not capture self-employment that was engendered by Micro Finance Institution lending.
  • The project generated 10,000 micro-credits by Micro Finance Institution and 27 sub-loans by PCB compared to an end target of 1000 loans and micro-credits. While this seems as a substantial achievement, the ICR (p. 25) pointed out that the end target was set very low.
  • The project improved access of rural households to rural agricultural credit from a baseline of 28% to 41.4% compared to an end target of 35%.
  • The project provided equipment and technical assistance for ensuring the connectivity, interoperability and integration of data management systems of the National Agency for Property Registration and its regional centers. Project activities also served as a catalyst for other donors (USAID, German Organization for Technical Development, Kreditanstalt für Wiederaufbau, Swedish International Development Cooperation Agency, and National Agency for Property Registration own resources) for the establishment of a network of 68 territorial centers.
  • The project supported the establishment of a Continuously Operated Reference System for the National Agency for Property Registration. The system was fully operational by project completion. By project completion the National Agency for Property Registration was a modern functioning and transparent cadastre and land registration agency.
Outcomes:
  • The project improved access to rural agriculture credit through strengthening the delivery of rural financial services and financial intermediaries. The project also strengthened property rights through capacity building of the National Agency for Property Registration. However, the ICR did not provide information regarding the development objective of improving agricultural production for Georgia’s small and medium-scale farmers and rural enterprises supported by the project. There is also a lack of evidence concerning the enhanced competitiveness of selected supply chains since there was only limited implementation of component 1, while the competitive grant program was never implemented. Finally, there was no information on adoption rates of new technologies or an assessment of the impact of new technologies on yields and incomes of farmers who adopted the agronomic advice received (ICR, p. 42).
(ii) to improve access to markets for Georgia’s small and medium-scale farmers and rural enterprises supported by the project: Modest.
Outputs:
  • The project strengthened the capacity of the food safety system through the rehabilitation and equipping of the food safety laboratory. Also, six regional centers for the National Service for Veterinary and Food Safety were constructed. However, the ICR (p. 25) pointed out that some of the laboratory equipment was not in operation at project closure and the territorial offices were not functional due to lack of equipment and furniture.
  • The project also supported training for the National Service for Veterinary and Food Safety staff in Latvia on the following topics: on-farm quality assurance for raw materials; controls of food and animal origin; Hazard Analysis Critical Control Points; Food and Hygiene Controls; EU Food Standards; EU Food Legislation; animal welfare; plant health controls; risk analyses (assessment, management and communication); control of food and feed of animal and non-animal origin.
Outcomes:
  • The project efforts supported some of the elements necessary for increased market access. According to information subsequently provided by the project team, the project was “transformational for the hazelnut value chain, facilitating the export of Georgian hazelnuts to the [European Union], amongst others, including supplying Ferrero, consistently ranked as one of the world's most reputable companies by Forbes' magazine.” The project team also reported that “project activities to improve the citrus supply chain led to exports to countries in the former Soviet Union, such as Ukraine and Kazakhstan.” According to the team, these achievements “‘directly result from the project and continue to yield positive results.” However, no further evidence is furnished of improved access to markets or the sustainability of such access. In addition, efforts to penetrate foreign markets were still incomplete at project closure, since the food safety laboratory was not operational at that time. The project team did not state that it had become operational since.

5. Efficiency:

The PAD (Annex 9) included a financial analysis of key potential crops and livestock in the project area (vineyards, apple orchards, hazelnut orchards, tea, greenhouse vegetables and pasture fed dairy cows). The analysis assumed that farmers would borrow no more than 80 percent of total investment at the prevailing interest rate (which was about 20 percent in June 2004) and would have grace periods of one to two years and repayment periods of one to four years depending on the investment. For each production system, net annual income after loan repayment, the financial internal rate of return and net present value were calculated. The opportunity cost of capital was set at 12 percent. The financial internal rate of returns were: 12% for dairy cows, 15% for hazelnut orchards, 20% for vineyards, 26% for apple orchards, 46% for greenhouse vegetables, and negligible for tea. No economic rate of return was calculated.
The ICR did not include an ex-post economic analysis for any of the project components. According to the ICR (p. 42) activities under component 1 had unquantifiable intrinsic benefits. In addition, critical data that should have been collected during implementation such as adoption rates of knowledge and information disseminated to farmers was not estimated during project implementation. There was also no assessment of the impact on yields and incomes of farmers who adopted the agronomic advice. For component 2, which received nearly 80% of all project funds, the ICR highlighted analytical limitations (such as size of investment, maturity and lending terms) that it considered an impediment to comparisons between sub-loans and the production models considered in the PAD. The ICR (p. 43) therefore cited the rate of return estimations included in the business plan which informed the lending decision by commercial banks as a proxy indicator for the financial performance of project supported investments. The median rate of return for a pool of loans was around 35%. For component 3, it was expected that building institutional capacity would generate several distinct types of benefits that could not be quantified due to early stage of roll-out (ICR, p. 44). The ICR provided no information on the investments in food safety institutions and property registration. However, it is unclear why an economic analysis of a random sample of the sub-loans provided under component 2 could not have been performed. Also, some measure of cost effectiveness for rehabilitation and equipping of labs could have been provided

There were some indications of operational and administrative inefficiencies. Implementation was undermined by delays in approving key project-specific documents, such as rural credit guidelines for commercial banks and non-bank financial institutions through the Ministry of Finance. There were also delays in making critical legal and institutional adjustments relating to credit unions and food safety that were essential for efficient implementation of project activities (ICR, p. 31). The Government’s decision to dissolve the Project Implementation Unit early in 2011 adversely affected the pace of implementation in the critical last few months of the project which resulted in several activities not being completed and the cancellation of nearly two-thirds of the PHRD funds.

Efficiency is rated modest.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Original objectives. Relevance of the original objectives is rated substantial while relevance of design, which was complex and lacking in focus, is rated modest. Efficacy was rated modest for both objective since little information was provided on the impact of the project on agricultural sales, enterprise profits and farmers' net income. Efficiency is rated modest. There was no ex post economic or financial analysis and no measures of cost effectiveness for rehabilitation and equipping of laboratories. There were a number of operational and administrative inefficiencies. Outcome for the original objectives is rated moderately unsatisfactory.

Revised objectives. Relevance of both objectives and design is rated substantial. The revised design reflected a better focus on improving agricultural productivity. Efficacy is rated modest for both objectives. Although the project succeeded in improving access to rural credit and increasing farmers' income, there is no evidence that the development objective of improved agricultural production for Georgia’s small and medium-scale farmers and rural enterprises was attained. Concerning the second objective, the project supported efforts to strengthen the capacity of the National Service for Veterinary and Food Service and rehabilitate the food safety laboratory, but these were incomplete at project closure. There is some evidence indicating improved access to markets, but it unclear whether this can be sustained. Efficiency is rated modest. Outcome for revised objectives is rated moderately unsatisfactory.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

Risk to development outcome is rated significant.
  • The absence of a public or private extension service in the country that could effectively disseminate knowledge created through project supported activities to a larger audience of farmers risks undermining the outcome of the project. Mitigation of this risk would require continuing donor and active private sector support for facilitating farmer access to integrated supply chains.
  • There is a risk that commercial banks will once again become averse to rural lending leading to a loss of institutional memory knowledge built with project support.
  • Non-bank financial institutions could face difficulties in securing external funding sources to sustain their lending activities to rural clients, although, according to the ICR (p. 29), such institutions may be in a stronger position to secure external lending given their established record of successful performance under the project.
  • There are sustainability concerns regarding the funding necessary for operation and maintenance of the project supported institutions, mainly the National Agency for the Public Registry and food safety institutions. This risk may be mitigated through Georgia's wish to comply with EU safety regulations in the context of a possible free trade agreement.

    a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:
Quality at entry is rated moderately unsatisfactory.

  • The Bank identified an appropriate intervention given the sector context. The project's objectives, although imprecisely formulated, were substantially relevant, and the thematic coverage and mix of components generally appropriate.
  • Component 2 – “Rural Finance Services” – was well designed and had a significant impact on rural finance, in particular in the non-bank financial sector.
  • In general, however, project design suffered from a weak causality chain in its results framework and a lack of focus, especially for components 1 and 3. Moreover, the original design was complex with nine sub-components. Some of the activities under component 1 involved novel approaches for Georgia and a considerable number of organizations were to participate in implementation. It is unclear that the implications of these factors for capacity building and the project's time frame were sufficiently taken into account.
  • The ICR (p. 29) acknowledged that some elements for components 1 and 3 were under-designed such as the Agriculture Supply Chain Development Fund, or lacked important implementation details as in the case of the food safety agenda. As the project team acknowledges, these components were not ready for implementation and were unfocused. Such design shortcomings hindered implementation of activities and eventually necessitated a significant revision through a Level 1 restructuring.
  • Although pest management issues were clearly relevant under component 1, the proper safeguard was not triggered due to the lack of coverage of pest management in the environmental analysis carried out during preparation (ICR p. 29).
  • The analysis of efficiency at appraisal was deficient as it only included a financial analysis (ICR, p. 29).
  • M&E design suffered from a lack of structure and specificity, particularly for components 1 and 3 (see Section 10 a below).

  • Quality-at-Entry Rating: Moderately Unsatisfactory

    b. Quality of supervision:
    Quality of supervision is rated moderately satisfactory.

  • The Bank maintained focus on achieving project objectives with the task team closely supervising implementation. According to the ICR (p. 30), the team maintained good rapport with other donors, especially IFAD, which resulted in an effective participatory approach in managing project activities.
  • In general, the team addressed issues in a timely manner and maintained a constructive dialogue between the project's major stakeholders. Issues raised during the mid-term review were followed by specific measures on the Bank side to facilitate the changes. The task team focused on implementing activities in line with the provisions of the environmental safeguards triggered by the project. These included Pest Management (OP 4.09) which was, for practical purposes, triggered for the period following the Level 1 restructuring.
  • However, the Bank could have been more pro-active in speeding up preparation of the Level 1 restructuring, which was approved 15 months after the mid-term review. The ICR (p. 30) acknowledged that design shortcomings and the wavering commitment of the Ministry of Agriculture should have been addressed earlier rather than after four years into implementation. Changes in the results framework were not incorporated in the legal agreement.

  • Quality of Supervision Rating: Moderately Satisfactory

    Overall Bank Performance Rating: Moderately Unsatisfactory

    9. Assessment of Borrower Performance:

    a. Government Performance:
    Government performance is rated moderately unsatisfactory.

  • The Government supported the project at the preparation stage. However, implementation was undermined by delays in approving key project-specific documents, such as Rural Credit Guidelines for commercial banks and Non-Bank Financial Institutions through the Ministry of Finance; and the operating manual for the Agriculture Supply Chain Development Fund and the Competitive Grant Program through the Ministry of Agriculture (ICR, p. 31). There were also delays in making critical legal and institutional adjustments relating to credit unions and food safety that were essential for efficient implementation of project activities (ICR, p. 31).
  • After the mid-term review the Government's performance improved, especially for the second and third components of the project.
  • Less than half the planned counterpart funding was actually provided, but the ICR (p. 31) reports that this was adequate for implementation purposes.
  • The Government’s decision to dissolve the Project Implementation Unit early in 2011 adversely affected the pace of implementation in the last few months of the project which resulted in several activities not being completed and the cancellation of nearly two-thirds of the available PHRD funds.

  • Government Performance Rating: Moderately Unsatisfactory

    b. Implementing Agency Performance:
    Implementing Agency performance is rated moderately unsatisfactory.

  • The Ministry of Agriculture was the implementing agency for the project.
  • In general, the Ministry displayed weak commitment to the project activities and objectives (ICR, p. 31). There was indecision on activities under component 1 and slow resolution of implementation issues. The Ministry's decision to start the liquidation process of the Agriculture Development Projects Coordination Center adversely impacted implementation in 2011 with inability to finalize multiple activities due to cancellation by the Ministry or inability of the Project Coordination Center or Implementation Unit to effect necessary procurement.
  • Despite these issues, the ICR describes the performance of the Project Coordination Center and the Project Implementation Unit as "generally satisfactory" (page 31). Fiduciary, procurement and safeguards aspects were addressed in a timely manner, while financial management was "constantly rated highly satisfactory" in supervision reports.

  • Implementing Agency Performance Rating: Moderately Unsatisfactory

    Overall Borrower Performance Rating: Moderately Unsatisfactory

    10. M&E Design, Implementation, & Utilization:

    a. M&E Design:
    The PAD (Annex 3) included a number of outcome indicators that attempted to capture the achievement of objectives and monitor progress. The design envisioned in the PAD called for the establishment of a baseline study and developing plans for data format and collection. However, the ICR (pp. 17&18) correctly identified a number of structural and presentational shortcomings in the results framework: for example, the project was supposed to be implemented over 5 years yet the PAD results monitoring table (p. 40) only put intermediate targets over 4 years (ICR, p. 18). The ICR (p. 17) also listed several discrepancies and inconsistencies in the PAD: for example, the table on arrangements for results monitoring in Annex 3 listed only three outcome indicators compared to seven in the Results Framework; also the output indicators for each component differ between the two tables in number for all three components, and in substance for components 1 and 3. In addition, outcome and output indicators were not only numerous, but also difficult to measure and suffered from attribution flaws (ICR, p. 18). Further, the indicators in some cases lacked specificity and according to the ICR (p. 18) reflected neither the logic nor the nature of the project inputs. The planned baseline study was not completed (ICR, p. 18). There were no data collection plans developed and monitoring of data for components 1 and 3 was limited to outputs with limited information on outcomes (ICR. p. 19). Implementation of M&E activities was to be the responsibility of the Project Coordination Center and Project Implementation unit.

    b. M&E Implementation:
    In accordance with the way the M&E system had been designed, monitoring related to components 1 and 3 was limited to outputs. Results under component 2 were better monitored since there was a sound system for providing timely and accurate information on the financial situation of participating financial institutions through data collected from the National Bank of Georgia, participating commercial banks and non-bank financial institutions (ICR, p. 19). However, monitoring of credit line beneficiaries based on questionnaires developed with IFAD assistance was not implemented. Also, only two outcome indicators out of seven in the results framework and 3 results indicators out of 14 were used to track project progress in the Implementation Status and Results Reports prior to the Level 1 restructuring (ICR, p. 18). According to the ICR (p. 19) this was partially due to lack of progress under component 1 and partially to deficient administrative arrangements towards the closing of the IDA Credit and PHRD Grant financing. After the Level 1 restructuring, a monitoring plan was developed for components 1 and 3, but was never implemented. The ICR (p. 18) states that restructuring helped to simplify and improve the indicators. However, the quantitative targets for some revised indicators were set at a fairly low level (ICR, p. 11). The revised indicators did not fully capture the impact of the project activities on increasing agricultural productivity and improving access to markets. According to the ICR (p. 19), the Project Implementation Unit complied with the project reporting requirements, although progress reporting was filed semi- annually instead of quarterly and sometimes with delays. An end project impact evaluation was not carried out primarily due to confusion arising from the liquidation of the project implementation unit. (ICR, p. 19).

    a. M&E Utilization:
    The ICR (p. 19) reports that data collected were used to inform decision making with regard to financial intermediation (component 2). In addition, collected data was used in gauging the impact of the project on target groups which, according to the ICR (p. 19), informed the updates on the project's outcome and results indicators.
    The ICR does not provide information on the utilization of any data collected for components 1 and 3.

    M&E Quality Rating: Modest

    11. Other Issues:

    a. Safeguards:
    Environmental Assessment. The project was rated environmental category FI (Financial Intermediary) and, according to the PAD (p.78), only OP 4.01 (Environmental Assessment) was triggered. The ICR reports that an environmental management plan and environmental guidelines were prepared to comply with the requirement of OP 4.01. OP 4.09 (Pest Management) should also have been triggered given that the implementation of the competitive grant program would have led to an increase in pesticide and fertilizer applications (ICR, p. 20). However, as the grant program was not implemented, this safeguard was only triggered after the Level 1 restructuring. From then on, environmental guidelines were updated to include a pest management plan and a pest management handbook (ICR, p. 20). According to the ICR (p. 20) there were no violations of OP 4.09 given the lack of implementation progress of the original activities of the sub-component. The ICR also reports that Bank safeguard policies were complied with throughout implementation. The overall safeguard and environmental assessment compliance was rated satisfactory in all of the project's Implementation Status and Results Reports except in two cases where the rating was moderately satisfactory. These latter related to: (i) the need for better compliance of the participating banks with the requirements of environmental screening and classification of sub-loans; and (ii) the need to ensure more individualized environmental management plans for construction works supported by the project. In both instances corrective measures were taken and ratings were subsequently upgraded. Pest management compliance (OP4.09) was rated satisfactory throughout the period following restructuring (ICR, p. 20).

    b. Fiduciary Compliance:
    Financial Management. According to the ICR (p. 19), financial management reporting was carried out at a high level of competence and financial management reviews of the Projects Coordination Center/Project Implementation Unit carried out by the Bank team were highly satisfactory. Annual external project audits were unqualified and their recommendations were addressed in a timely manner.

    Procurement. According to the ICR (p. 20), procurement activities were consistently executed in line with World Bank policies and procedures, and ratings were never rated lower than moderately satisfactory. Post Procurement Reviews generally reflected reliable, timely and transparent procurement activities (ICR p. 20).

    Disbursement. Disbursements were slow at the beginning of project implementation. The reallocation of proceeds after the Level 1 restructuring on July 9, 2009 from supply chain development (component 1) towards rural financial services (component 2) was accompanied by a surge in disbursements (ICR, p. 20). By project completion, 100% of the IDA credit, 88% of IFAD and 36% of PHRD funds had been disbursed. The poor disbursement registered for PHRD grants was due to the lack implementation progress under component 1.

    c. Unintended Impacts (positive or negative):
    None reported.

    d. Other:
    None.



    12. Ratings:

    ICR
    IEG Review
    Reason for Disagreement/Comments
    Outcome:
    Moderately Satisfactory
    Moderately Unsatisfactory
    There is insufficient evidence of increased agricultural productivity, profitability, production, or sustained enhanced market access as a result of project-supported activities. The absence of any meaningful ex-post economic analysis makes it difficult to assess efficiency. 
    Risk to Development Outcome:
    Significant
    Significant
     
    Bank Performance:
    Moderately Satisfactory
    Moderately Unsatisfactory
    There were significant shortcomings in quality at entry (see Section 8a above). Quality of supervision is moderately satisfactory. Since outcome is assessed as moderately unsatisfactory, overall Bank performance is moderately unsatisfactory in accordance with the Joint IEG/OPCS Harmonization Criteria. 
    Borrower Performance:
    Moderately Unsatisfactory
    Moderately Unsatisfactory
     
    Quality of ICR:
     
    Satisfactory
     
    NOTES:
    - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
    - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

    13. Lessons:
    The following lessons are taken from the ICR with some adaptation of language:
    • The project would have benefited from more thematic focus, and subsequently fewer, yet more easily implementable activities. Adequate balance is always necessary between a meaningful and manageable scope and the instinct to go for more. Ideally, project-supported themes should be complementary and provide for mutually enabling, and perpetuating results. The project would have benefited from fewer components and sub-components (activities). This would have also contributed to a more precise initial formulation of the development objective and to a tighter and more logical results framework.
    • A robust M&E framework developed during preparation, with sound connections between the variables to be monitored and the project’s objectives, is a very useful tool in supervising implementation both by the Bank and by the Borrower. The experience of this project demonstrates the undesirable consequences of leaving the conceptualization of M&E arrangements for the early stages of project implementation.
    • Realism. Overall project design and activities that are grounded in the institutional reality of a country will facilitate implementation. Setting realistic expectations in areas of policy reforms, regulations and institutional strengthening is a prerequisite for successful and timely implementation of projects that address institutional modernization.
    • Pro-activity in restructuring problem projects. In order to make a meaningful difference to outcomes, substantive corrective measures, such as a Level 1 restructuring, need to be undertaken sufficiently early in the implementation cycle.

    14. Assessment Recommended?

    Yes
    Why?
    To verify the ratings and document lessons learned.

    15. Comments on Quality of ICR:

    Overall, the quality of the ICR is rated satisfactory. It provides a thorough yet concise account of project activities and a candid view of most shortcomings and also provides some valuable lessons for future projects. There are, however, some shortcomings. The discussion on the achievement of the development objectives is limited. In addition, the ICR did not include an ex post economic or financial analysis of the project activities which would have provided a better assessment of efficiency. Finally, the weighted outcome rating in the ICR is calculated erroneously based on a disbursement of 44% at the time of restructuring rather than 58%.

    a. Quality of ICR Rating: Satisfactory

    (ICRR-Rev6INV-Jun-2011)
    © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions