Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Agricultural Competitiveness Project


  
1. Project Data:   
ICR Review Date Posted:
02/28/2014   
Country:
Kazakhstan
PROJ ID:
P049721
Appraisal
Actual
Project Name:
Agricultural Competitiveness Project
Project Costs(US $M)
 70.8  46.0
L/C Number:
L4774
Loan/Credit (US $M)
 24  14.7
Sector Board:
Agriculture and Rural Development
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  04/28/2005
 
 
Closing Date
07/30/2010 06/30/2012
Sector(s):
Agricultural extension and research (50%), Agro-industry marketing and trade (18%), Animal production (13%), Crops (12%), Central government administration (7%)
Theme(s):
Rural services and infrastructure (29% - P) Export development and competitiveness (29% - P) Other rural development (14% - S) Rural policies and institutions (14% - S) Rural markets (14% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Ebru Karamete
Ridley Nelson Christopher David Nelson IEGPS1

2. Project Objectives and Components:

a. Objectives:
The Loan Agreement (p. 12) statement of the development objective is:

“to increase the competitiveness of the agricultural sector by facilitating access to markets and knowledge by: (i) improving the quality and safety of agricultural products, access to information and market efficiency; and (ii) increasing the quality, quantity and relevance of public and private investments in applied agricultural research and extension and knowledge transfer."

The project development objective stated in the Project Appraisal Document (PAD, p. 6) is:


"to increase the competitiveness of the agricultural sector in Kazakhstan. To achieve this objective, the project would facilitate access to markets by supporting measures to improve the quality and safety of agricultural products, enhance access to information, and harmonize standards. It will also help to increase the quality, quantity, and relevance of applied agricultural research and facilitate transfer of knowledge to farmers".


As per IEG’s current practice, this Review’s assessment is based upon the formulation of the project objective as in the Loan Agreement.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 10/28/2011

c. Components:
1. Quality and Safety Management of Agricultural Products (appraisal estimate US$ 31.0 million, actual US$ 10.4 million).
This component aimed to enhance the management of food safety controls and quality certification along the value chain. Harmonization and development of standards sub-component supported the country’s ongoing efforts to harmonize standards by (i) establishing technical committees on harmonization of regulations and standards related to agricultural products; (ii) providing training on technical regulations and standards; (iii) financing an awareness campaign aimed at generating interest in the work of the committees and disseminating their achievements; (iv) supporting efforts to monitor and certify organic production in accordance with internationally recognized standards. Quality and safety monitoring sub-component tried to strengthen the capacity of public and private entities to monitor food quality and certify standards of agricultural products through an internationally recognized system for testing and monitoring of quality and safety. Project funds supported establishing and equipping a veterinarian and a plant protection testing center, modernizing laboratories for testing seeds and inputs, providing training and financial incentives (matching grants) to encourage public and private laboratories to seek accreditation and implementing quality assurance schemes in selected agro-enterprises. The construction of a National Reference Laboratory under this Component was canceled in 2011 at the request of the government due to a complicated tender process and delays in implementation.

2. Agricultural Marketing (appraisal estimate US$ 4.4 million, actual US$ 3.5 million).

This component aimed to enhance agricultural producers’ and processors’ understanding of markets, improve marketing infrastructure, and facilitate equal access to market information. Strengthening the Market Information System Sub-component tried to enhance the existing system by improving the quality and frequency of data reporting, using enhanced media for publishing information, and issuing analytical reports. Developing Market-oriented Infrastructure Sub-component provided financial incentives to develop marketing associations or partnerships, or both. Eligible sub-projects for up to 40% co-financing included facilities such as milk collection points, slaughterhouses, storage facilities, and distribution networks for processing of priority commodities. According to the ICR, (p. 4) grants were for projects with a technical assistance component, an innovative aspect, a plan for disseminating results, or public benefits beyond the farm. Market information sub-component was cancelled in 2009 during the streamlining effort of the project.

3. Applied Agricultural Research and Extension (appraisal estimate US$ 28.2 million, actual S$ 27.6 million).

This component aimed to increase the effectiveness of agricultural research and extension services in Kazakhstan by facilitating the adoption of innovations that increase the productivity of farmers and agro-processors. Specifically it tried to create a competitive funding mechanism for applied research and extension and to create a structure for training and supporting new extension agents. Through this component the Ministry of Agriculture tried to considerably strengthen extension services in rural areas, by employing at least one extension agent per district and one extension supervisor per oblast. Project funds were planned to be used to train and certify 400 private extension agents as well as finance around 450 extension and training projects through the Competitive Grant program.

4. Institutional Development and Agricultural Policy (appraisal estimate US$ 6.2 million, actual US$ 4.5 million).

This component aimed to create the institutional structure to implement project activities and to help the Ministry of Agriculture to establish the policy and institutional framework to improve the competitiveness of the country’s agricultural sector. It comprised three sub-components: institutional structure, project evaluation, and agricultural policy development. The institutional structure sub-component tried help the government to separate roles between policy making, implementation, and technical review by supporting the establishment o f a Governing Board, Coordination Center and the roster of independent peer reviewers, each with distinct roles and responsibilities. The governing Board would be responsible for defining the strategic guidelines, including funding systems under the three components. The Project Coordination Center would act as secretariat of the project and be responsible for implementing the policies. The rotating roster of independent national and international experts would be responsible for the technical review of proposals submitted under the Competitive Grants Scheme. The project evaluation sub-component aimed to finance the technical assistance to monitor and evaluate project implementation and outcomes. Agricultural policy development subcomponent tried strengthening the capacity of government to analyze, formulate, and monitor agricultural policies; it also financed training, for selected staff Ministry of Agriculture. Component 4.2 (project evaluation) was cancelled in 2009 during the streamlining effort of the project.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Costs:
Total costs were revised on October 28, 2011, from the appraisal estimate of US$ 83.1 million to US$ 64.9 million. The actual total spending was US$ 58.3 million due to cancellation of several project sub-components (Component 1.1, 2.1, 4.2), streamlining of project activities, as well as discontinuing national reference laboratory due to issues with its tender process, appraisal estimate was reduced.


Financing: During the project restructuring in 2011, US9.3 million of the US$24 million Bank loan was cancelled because of discontinuation of national reference laboratory activity At project closing, 61 % of the original loan was disbursed. Local beneficiaries were expected to contribute US$ 12.3 million during appraisal and a 100 % of this amount was contributed by project closing.

Borrower Contribution:
It was estimated that the Borrower contribution would be US$ 46.8 million at appraisal, this amount was reduced to US$ 38 million with the 2011 restructuring and at closing 82 % of this revised amount was contributed by the Borrower.


Dates:
The loan closing date was extended from July 31, 2010 to June 30, 2012 following delays due to the long ratification by the Parliament of project start up and delays in construction of the National Reference Laboratory (under part A2 of project description of the Loan Agreement). On October 28, 2011 another restructuring was made with partial cancelation of the loan amount; amending the loan agreement by deleting reference to National reference Laboratory as well as revising the monitoring and evaluation framework to reflect changes in the project costs and financing to remove discrepancies with the Project Appraisal Document
.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Substantial
The project development objectives of increasing agricultural competitiveness through access to markets and knowledge was relevant at the time of appraisal and is still relevant to the country priorities. In the mid 2000s, agriculture had significant potential to help accelerate the country’s growth contributing, 8 % of GDP and employing 32 % of the economically active population (ICR p. 1). However, both crop yields and livestock productivity were well below levels reached in countries with similar agro-ecological conditions. For example crop yield was approximately one third of Canada and milk yield was one third of New Zealand (PAD page 2). In order to unleash the potential of agriculture, competitiveness of the agricultural sector had to be improved through access to markets, know-how and technology and the appropriate financial services to serve small farmers. Due to agricultural sector’s limited access to international markets, finding new export markets for agricultural products was key to be able to contribute to economic growth. During that time, Kazakhstan had applied for membership in the World Trade Organization and as part of the Sanitary and Phytosanitary Agreement, it had to harmonize animal and plant health standards. Also, private enterprises were having difficulties in implementing private standards to meet customer demand. Another issue was that small farmers who had increased in number since the transition and therefore had started to have a larger share in production (producing more than a third of grain, over 50 percent of meat, and more than two thirds of raw cotton) (ICR page 1), had limited access to knowledge and technology. At the same time, the system of research, technological development, and extension that served farmers during the Soviet era collapsed and a new system to meet the needs of increasing numbers of small farmers had not been developed. These issues had to be addressed in order to increase competitiveness of agricultural sector.

The project objective was relevant to the government’s Agro-Food and the Rural Development Program that aimed to stimulate agricultural growth and promote rural development via increased budget allocation, institutional reform and improved legal framework. The Ministry of Agriculture had increased its oversight of management of natural and applied agricultural research. New laws to encourage agricultural growth and rural development, including land, forest and water codes, the microfinance law, and the law on credit partnership, among others had been passed. The land code allowed private ownership of agricultural land, which was critical for overall agricultural development. (PAD page 5).

The project objective was also relevant to FY 2012-2017 Country Partnership Strategy, specifically the second outcome, “expanding non-oil sector exports and employment”; and the fifth outcome, “strengthening knowledge for sustained growth in agriculture”.

b. Relevance of Design:
Modest

The design had a significant shortcoming. The link between objective, outcomes, outputs and the funding was not clear. The objective of increasing the competitiveness of the agriculture sector, was ambitious, broad and ambiguous and, partly as a result, it was unclear how it would be measured. The PDO did not define what aspects of competitiveness were intended by the project such as increased quality of specified products and increased market share in selected markets, making any focused measurement consistent with the Results Framework difficult. It was not realistic to expect a small project to increase the competitiveness of the entire agriculture sector. Enhancing the management of food safety controls and quality certification (Component 1), improving market access (Component 2) and increasing the effectiveness of agricultural research and extension services and thus increasing productivity or reducing costs (Component 3) may all be factors in increasing competitiveness but they could not alone enhance the competitiveness of the sector. The Competitive Grant Scheme, one of the key activities of the project, could only fund 572 sub-projects, small in relation to the size of agriculture in the country.


4. Achievement of Objectives (Efficacy) :

The objective was to increase the competitiveness of the agricultural sector. The intermediate objectives were: (i) improving the quality and safety of agricultural products; (ii) improving access to information; (iii) improving market efficiency; (iv) increasing the quality, quantity and relevance of public and private investments in applied agricultural research and extension and knowledge transfer. Below the four intermediate objectives are covered first followed by the overarching objective.

Intermediate Objectives:


(i) Improving the quality and safety of agricultural products. Rated modest.

Outputs:

7 technical regulations on quality and safety of food products were developed and adopted (for selected key commodities including meat and milk products, fruits and vegetables and grains). The output was in line with the output target of 7 regulations.

34 state food safety standards were harmonized.

6 guidelines for the introduction of food products safety management systems were written.

23 private food-processing companies received grants for the introduction of the food products safety management systems.

6 types of lab equipment were delivered with project support into 18 oblast and 60 rayon branches of the Republican Veterinary Laboratory and 4 oblast phytosanitary labs. The original output target was obtaining accreditation of 60 laboratories as complying with international standards, and this was found to be unrealistic and the indicator was modified to “laboratories equipped according to international accreditation standards”. However, a needs assessment for laboratories was not conducted during preparation, and the list of equipment was not tailored to the needs of each oblast, therefore in many cases equipment purchased were under-used.

Also the construction of a National Reference Laboratory under Component 1 was canceled at the request of the Government due to much higher cost of construction than envisaged and a complicated tender process.

Outcomes:

The PDO indicator set by the Project was, “increase the proportion of agricultural products that are tested and meet international standards for quality and safety”. The baseline for this indicator was 10 % of products tested and end of project target was 30 % of products. At project closing, only about 15 % of products tested met international quality and safety standards or about 25% of the increment targeted.

There is no other evidence whether quality and safety of products improved.

(ii) Improving access to information. Rated substantial.

Outputs:

A state extension system was created and supported. This could be considered an intermediate outcome.

2,500 farmers and entrepreneurs received basic knowledge on agricultural marketing, and marketing information system.

9 training centers all equipped with call centers in agriculturally important locations were established.

Extension training centers supported under the project provided training to about 7,800 farmers.

2,828 contracts were signed with farmers for subscription service whereby farmers received a minimum package of consultation and information.

Database of agricultural producers and their main products was developed registering 56,807 producers, as well as a database containing information on 3,014 research results was created.

Strengthening Market Information System under Component 2 was canceled.

Outcomes:

There was no specific outcome indicator to measure access; however the output data suggest that, through the creation of state and private extension services, many farmers accessed information and know how in the form. of training and consultation. However, market information enhancement was not achieved.

(iii) Improving market efficiency. Rated modest.

Outputs:

160 various scientists and specialists received training abroad on agricultural marketing and competitiveness.

2 training modules were prepared on the development of marketing and information system and training was provided to 570 farmers, as well as staff members of government agencies.

170 training sessions on marketing were held in total.

2,500 farmers and entrepreneurs received basic knowledge on agricultural marketing, and marketing information system

77 competitive grants for developing marketing infrastructure and improving the image of agricultural products were awarded. This was below the target of 140 market oriented sub-projects. One specific example was development of Kaznan brand to increase export potential of flour and pasta products through a project grant.

Outcomes:

The project set the following outcome indicator to measure market efficiency: “increase the value of agricultural exports, including livestock products, compared with 2003”. The value of agricultural exports increased by 130 % between 2005 and 2011. However, attributing this outcome entirely to the project is not possible. Furthermore, as reported in the ICR (p. 14), world food prices increased 98 % during the same period. So, this increase was primarily due to food price increases. There is no other evidence provided by the ICR that documents an improvement in market efficiency.

(iv) Increasing the quality, quantity and relevance of public and private investments in applied agricultural research and extension and knowledge transfer. Rated substantial.

Outputs:

A state extension system was created and supported provided by a newly established holding, Kazagroinnovation which united all agricultural research institutes in the country.

9 training centers all equipped with call centers in agriculturally important locations were established.

Extension training centers supported under the project provided training to about 7,800 farmers.

2828 contracts were signed with farmers for subscription service whereby farmers received a minimum package of consultation and information.

472 The Competitive Grant Scheme financed applied agricultural research (test of new varieties and breeds), extension, and technology introduction. This was below the target of 600 applied research and extension sub-projects.

27 sub-projects by Kazakh National Agrarian University for testing and adopting new technologies in crop and animal sciences were funded by Competitive Grant Scheme.

Project supported Ministry of Agriculture in mandating “Joint Stock Company Kazagroinnovation” to establish an outreach network at regional and district levels for extension services.

Outcomes:

There is no specific outcome indicator to measure this intermediate objective. However the project helped to build the country’s agricultural extension system, which is a potentially positive means for knowledge transfer. The Project team reported that the system is effective and sustainable. Kazagroinnovation conducted satisfaction surveys of 857 beneficiaries, which showed that 71 % of participants had used the technical skills acquired in Kazagroinnovation’s training events (ICR p. 14).

Involvement of academicians/universities via the competitive grant program was effective to test and accelerate adoption of new technologies.

The ICR reports that the project fostered cooperation between local and central governments, academia and producers.

The strategy of conducting research on beneficiary farms rather than on research stations is reported to have improved the extension services and the impact of sub-projects. While no direct evidence on this particular practce is cited, experience suggests that this is plausible.

Main Objective:

Increasing the competitiveness of the agricultural sector. Rated modest

Outputs:

Through the Competitive Grant Program 572 activities were financed (in the total amount of US$20.3 million equivalent) . These are reported to have enabled companies to improve their competitiveness by the following means: modernization of private laboratories, development of agricultural marketing infrastructure (slaughterhouses, storage facilities), image enhancement of agricultural products (rebranding, packaging), applied research (test new varieties and breeds), extension and demonstration. However, the achievement in terms of number of grants was lower than the outcome target of 800 activities. Of the wide spectrum of technologies promoted by the project, conservation agriculture (no-tillage technologies), adoption of new wheat varieties, testing new methods on animal disease control, or irrigation technologies such as drip irrigation are reported to have been promising, however, apart from the zero tillage that was widely adopted nationally there is no data on the actual usage of these technologies.

Number of Grants by Type PlannedActual
Laboratories (private)2025
Market-oriented infrastructure14058
Image enhancement3019
Applied research10098
Extension and demonstration430372
Total800572

Outcomes:

The main outcome indicator was: “increasing farm income particularly for small and medium size farmers”. An estimated 3,200 farmers increased their incomes as a result of the project. The original target was 2 % of farmers and this was revised to 2.400 farmers to better reflect Project’s contribution. Whether this constitutes much change is not clear since 2% of the medium and large farms would be about 2,500. There was no data on whether the project contributed to an income increase of small and medium size farmers in general in the country. Also, as mentioned in section 3.b, this indicator is not a very adequate measure of increased competitiveness, increased profitability and/or increased yields could be better measures of competitiveness.

In fact, profitability was originally included in the results framework as a measure of competitiveness, but was dropped in 2011. The indicator was monitored until 2010 and 63.9 % of peasant and family farms were profitable at that time compared to 37.2 % in 2004. This exceeded the target of 50 % of farms covered by the project.

The ICR reported that according to survey data 30 % of producers adopted new technologies (p.26), presumably because they found them at least potentially profitable.

Croplands under zero-tillage increased from zero in 2001 to 500,000 in 2007 and to 1.85 million hectares in 2012, a large increase that might plausibly contribute to both efficiency and environmental gains and higher incomes for farmers. However, the attribution to the project of these national numbers is unclear.

The project did contribute to overall sector development through the adoption of technical regulations, upgrading the country's applied research capacity, establishing an efficient extension capacity as well as helping 572 businesses to become more competitive through the grant program. However, given the size of the agricultural sector versus the project size, achieving the main outcome of increasing the competitiveness of the agricultural sector was highly ambitious. Also, as mentioned in Section 3.b, the outcome indicators did not adequately measure improved competitiveness of the sector. Some outcome indicators such as ‘increase in agriculture exports’ could not be attributable to the Project entirely.

Finally, two important activities that could have potentially contributed to achieving the objective were either not implemented (National Reference Laboratory), or would require further assistance with its optimization (laboratory upgrade program for animal health and food safety).

There is limited evidence on achievement of outcomes in terms of competitiveness. However, the achievement of two important intermediate outcomes - setting up an extension system that covered the entire country, and upgrading the Kazakhstan’s research capacity - would both be expected to be necessary conditions for achieving the competitiveness objective. Balancing the lack of competitiveness evidence with the accomplishments and expectations of the impact of research and extension in the longer term as improved technologies work their way through the system, the rating of the achievement of this overarching objective is modest.


5. Efficiency:

Efficiency is rated modest.

The PAD presented an economic analysis (PAD, pages 12-13) with the following assumptions, the benefit stream derived from: (a) the improved quality and safety of agricultural products, as well as the improved market environment at the national level; and (b) improved effectiveness and profitability at the farmers’ and rural entrepreneurs’ level. Regarding the former element, wheat was taken as a proxy as a major crop. At the time of appraisal average annual export of wheat was 5 million tons. It was estimated that as a result of project investments in quality management, quality of wheat would gradually increase and at project end (year 5) 10 % of exports or 0.5 million ton wheat would graduate from class 3 to class 2, and at full development (year 7) 20 % or 1 million ton would graduate from class 3 to class 2. The difference in price between wheat class 3 and 2 was US$15. Based on these assumptions, the annual incremental benefits from the first benefit stream element at year 7 were estimated at US$15 million. In regard to the second benefit stream, a total number of seven activity models were prepared (laboratory investment, slaughterhouse, milk collection center, applied research on improved fertiliziation of wheat, applied research improved feeding for cows, extension services- increased technology for cotton, extension services-good agricultural practices for soybean). (Given the demand-driven aspect of the Competitive Grant Scheme specific activities were estimated.) The following incremental annual net benefits per one dollar of investments were estimated: Laboratory equipment 0.22; Marketing development 0.24; Applied research 0.94, Extension 0.62. The derivation of these is not entirely clear. The models showed that the activities in applied research and extension have an ERR in a range of 16 % to 80 %. According to these assumptions, the base case internal rate of return for the total project was estimated in the PAD at 23.5 %. The base case net present value of the project’s net benefit stream, discounted at 12 percent, was USS 51 million. According to the sensitivity analysis, a fall in total project benefits by 20 % and an increase in total project costs by the same proportion would reduce the base economic rate of return to about 17 %, which was still above the discount rate of 12 %. However, it is not clear how the assumptions regarding increase in quality of wheat were calculated and if the project investments would lead to such an increase.


Different from the PAD analysis, the ICR economic analysis calculation considered only the benefits generated by the Competitive Grants Scheme. The ICR reported that (p.30) due to lack of reliable data, benefits on improved quality of wheat could not be estimated. The project team subsequently stated that the Competitive Grant Program, was the key quantifiable benefit resulting from project investments and was cutting across Components 1, 2, and 3 and representing 67% of the total project cost (US$31 million out of US$46 million). The appraisal also expected benefits from improved quality and safety of agricultural products and more effective marketing. However, these were difficult to estimate due to lack of reliable data. The economic rate of return and the net present value at completion were an estimated 29.2 % and US$ 70.6 million, respectively, assuming an opportunity cost of capital of 12 %. The ICR reported that (p.30): “..14 grant projects located in South-Kazakhstan, Almaty, Zhambul, Akmola, Kostanay, and Karagandy oblasts were visited and analyzed, or 2.5 percent of all projects implemented. This is considered a representative sample for the project”. However, this is a very small sample, particularly since there were a number of entirely different categories of investment giving, in some cases, one case per category. Moreover, it is not entirely clear how the random sampling was made. Another issue is that the analysis included benefits attributed to beneficiaries who adopted the promoted technologies without the project’s financial support; however no evidence was presented to show how these spillover benefits are linked to and therefore attributable to the project or the extent to which they were simply part of an already established national trend. There are, therefore, significant limitations in the efficiency evidence.

The analysis included projects in the following areas: modernization of private laboratories, development of agricultural marketing infrastructure (slaughterhouses, storage facilities), image enhancement of agricultural products (development of new design for packaging and websites, rebranding), applied agricultural research (test of new varieties, breeds, conservation agriculture), and extension and technology introduction. The benefits considered by the economic and financial analyses were higher incomes for farm and rural nonfarm enterprises resulting from project extension services on improved technologies on farming (including zero tillage) and agro-processing industries, as well as better marketing strategies. Accordingly, an average incremental annual net benefit of US$4 per US$1 of investment was estimated. A sensitivity analysis assessing the effect of variations in benefits and costs revealed that a fall of 20 % in total project benefits and an equivalent increase in total project costs would reduce the base case ERR to about 26 %, which was significantly higher than the opportunity cost of capital.

Project implementation was only efficient to a certain degree. As evident from the disbursement profile, loan funds started to disburse approximately 3 years after project approval. The project closing date had to be extended by 2 years due to delays in implementing approved competitive grant sub-projects as well as delays in implementing of the National Reference Laboratory construction. Main reasons for delays were: (i) the rigidity of government’s bureaucracy that made the implementation complex and slow; (ii) poorly drawn lines of responsibility among the ministries; (ii) weak capacity of the implementing agency and the change of management in the Ministry of Agriculture that resulted in critical shifts in project management particularly in terms of reorganizing the Project Coordination Center during the last year of implementation.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
23.5%
100%
ICR estimate:
Yes
29.2%
100%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The relevance of objectives is rated substantial. The objective to increase the competitiveness of the agricultural sector by facilitating access to markets and knowledge was consistent with the country priorities. The relevance of design is rated modest as the link between objectives and project activities and funding were weak. Under efficacy, the main objective of increasing the competitiveness of the agricultural sector is rated modest as there is not sufficient evidence to show that this was achieved, high weighting is given to this over-arching objective. The intermediate objective of improving the quality and safety of agricultural products is rated modest due to lack of evidence. The intermediate objective of improving access to information is rated substantial. The intermediate objective of improving market efficiency is rated modest again due to lack of evidence and the intermediate objective of increasing the quality, quantity and relevance of public and private investments in applied agricultural research and extension and knowledge transfer is rated substantial. Efficiency is rated modest mainly due to weak economic analysis methodology.

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

The ICR’s assessment of risk to development outcome was moderate. However, this review’s assessment of risk rating is higher. The development objective of increasing agricultural sector’s competitiveness could be affected by: (i) changes in the macroeconomic situation including overvalued currency and decline in agricultural exports; (ii) lack of improvements in business environment and transportation infrastructure that could jeopardize other measures towards increased competitiveness; (iii) better performance by competitor countries, which could negate achievements in competitiveness of Kazak agriculture. The “competitiveness” objective is an inherently risky objective, it is not only subject to macroeconomic conditions and additional measures on improved business environment and infrastructure, but also it is subject to what other competitors do, which is beyond national or project control.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:
The Bank provided substantial technical support for project preparation, and strong sector analytical work through background notes, economic sector work, and sub-sector studies on wheat and cotton. Lessons learned in other countries were also applied; some of the important ones were: (i) redefining the role of the state and of the private sector in agriculture by leaving the state with regulatory and policy making role; (ii) involving all stakeholders such as scientists, extension agents, farmers, NGOs, etc. in setting the research agenda and in designing extension services for maximum impact; (iii) encouraging innovation by offering matching grants through competitive process for applied and adaptive research and knowledge transfer. The technical design was influenced by the fact that government was more interested in obtaining knowledge and experience than project financing itself.

A significant weakness was in the results framework. As mentioned in section 3.b, the link between objective, outcomes, outputs and the funding was not clear, the main problem being the ambitious and very brood objective, that was not well aligned with project components.

Also, the project had a complex structure trying to achieve a large number of planned activities resulting in complicated implementation and coordination. Therefore, some activities that were unlikely to succeed such as MIS, image enhancement and policy development had to be eliminated. Furthermore, the project budget estimated at appraisal was inadequate for some key activities- particularly construction of the National reference laboratory; this led to major delays in procurement and eventually cancellation of this sub-component.

The institutional arrangements were generally sound with the Ministry of Agriculture responsible for overall project execution and an institutional structure for project implementation that would be created at the ministry, comprising a Coordination Center and the roster of independent peer reviewers (PAD p. 10). A governing board would be responsible for overseeing project implementation. The independent peer reviewers would be responsible for reviewing and selecting proposals for funding under the Competitive Grant Scheme.

The M&E design and institutional set up for M&E was weak (see Section 10).

The identification of project risks was comprehensive at appraisal, correctly focusing on the risks associated with cumbersome bureaucratic procedures. Other risks arising later, especially of complications in project implementation resulting from the 2005 budget code, could not have been anticipated at project appraisal.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:
According to the ICR (p.19) the World Bank team provided technical guidance through regular supervision missions and the aide memoirs, as well as constant supervision on implementation of fiduciary and safeguards policies. The following shortcomings were reported by the ICR: The team could have made an effort to drop the National Reference Laboratory construction activity at an earlier stage, as it was evident during the time of extension that the activity could not have been completed before project closing. Also, the team could have tried to redesign the laboratories sub-component into laboratory accreditation activity rather than just the procurement of laboratory equipment. This would more directly serve for the competitiveness objective due to the augmented product certification capacity to meet minimum animal and plant health standards. In addition to these shortcomings identified in the ICR, this review finds that the team did not try to revise the project development objective in order to address the unrealistic scope of the project and to revise the inadequate outcome indicators within the Results Framework in order to link them with the objective.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:
Government delays in meeting the conditions for loan effectiveness, had a negative effect on the project’s momentum from the start. Lengthy clearance process by the Ministry of Finance for every individual project activity (including for example approval of the consultants’ terms of references and purchase of computers for the Project Coordination Center) was another factor that negatively affected the pace of implementation in the early stages. However, these difficulties were resolved after streamlining the clearance process by eliminating several steps. This helped to accelerate project implementation, and the pace accelerated during the third year of implementation, but slackened thereafter.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:
The ICR reported that (p.20), the Ministry of Agriculture showed little ownership of project activities and particularly component 1 suffered from lack of leadership (e.g. oblast- and rayon-level laboratories complained about the lack of response from the Ministry when communicating their problems and needs). The Project Coordination Center was dissolved as of June 30, 2012, without any follow-up on the two activities that were still ongoing- the review of the laboratories and the collection of final reports and final payments to the 241 Competitive Grant Scheme beneficiaries whose sub-projects were not closed by June 30, 2012. Although it did not involve any IBRD loan funds, the fact that several grants awarded under the project will not be paid out in full violates the contractual agreement with the beneficiaries and poses reputational risk for both the World Bank and the government. Also, the National Reference Laboratory design contract was similarly terminated, wasting efforts done in this regard. Furthermore, significant delays in project implementation occurred because procurement activities took four to eight months from drafting to bidding. The legal department of the implementing agency, failed to provide constructive support and presented considerable bureaucratic impediments to implementation. The above evidence provided by the ICR indicates a moderately unsatisfactory rating.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
A major shortcoming of the M&E system was that it was not designed to measure the achievement of the project’s objective of increasing the competitiveness of the agricultural sector. M&E was designed to measure: (i) market access through changes in the value of exports; (ii) quality and food safety by the proportion of agricultural products tested and meeting international food safety standards; (iii) harmonization by the number of technical regulations harmonized; (iv) increasing the quality, quantity, and relevance of applied agricultural research by outputs instead of outcomes. Increasing the access to information was not measured. Also, none of the chosen indicators could directly inform the achievement of the project’s competitiveness objective. The institutional set up for M&E included Project Coordination Center as mainly responsible for carrying out M&E activities and also a company was hired to carry out Impact Analysis, but the performance was not satisfactory.

b. M&E Implementation:
Regarding M&E implementation, the outputs of project activities were not monitored on a regular basis. A consultancy firm was hired to evaluate project impacts, but it was not clear if this was for annual evaluation as planned in appraisal or a one time evaluation. The consultancy firm’s final impact assessment report was not of good quality as it did not review the project’s achievement of its objective, or other impacts. Therefore, additional resources had to be spent to assess the project’s impacts properly. The Project team reported that in order to compensate for the gaps on impact assessment, FAO was hired to carry out project impact assessment and their analysis was used to report final project results. It was reported by the ICR that (p.10) monitoring activities conducted by implementing agency Joint Stock Company Kazagroinnovation were better implemented. The company closely monitored the impacts of training and other services and reported results to the Ministry of Agriculture. As mentioned in Section 4, Kazagroinnovation conducted satisfaction surveys to assess outcome of its training activities.

a. M&E Utilization:
The limited and irregularly collected information was used to the extent possible to follow project implementation at the output level. There was limited use of M&E evidence at the outcome level.

M&E Quality Rating: Negligible

11. Other Issues:

a. Safeguards:
The project was under the environmental assessment category financial intermediaries (FI). Two safeguards policies were triggered under the project: OP 4.01 (Environmental Assessment), and OP 4.09 (Pest Management). According to the ICR (p.10) overall, the project was implemented in compliance with the Bank and national environmental assessment rules and procedures as well as existing environmental requirements. In order to safeguard against negative environmental impacts, manuals were developed on laboratory operations, and a special environmental review document was written specifying the environmental assessment rules and procedures to be applied for supported grants. Also training was provided in pest management. About 25 Competitive Grant Scheme sub-projects generated some adverse, but temporary and localized, environmental impacts, including waste water, emissions, solid waste, and soil erosion. The environmental permits and authorizations requested were disclosed on the project website, making this information available to all interested parties. The ICR did not present safeguard compliance ratings but according to the Project team report, the compliance was found to be satisfactory.

b. Fiduciary Compliance:
According to the ICR (p.11) overall the project was in compliance with the financial management covenants during implementation. The Financial Services Department of the Ministry of Agriculture and the Project Coordination Center were responsible for managing the project’s finances and those arrangements were reviewed regularly during project implementation and found to be generally acceptable to the Bank. The annual audit reports were found to be acceptable to the Bank and contained unqualified (clean) opinions. ICR also reported that (p. 11) ..”the Financial Services Department experienced problems in managing the designated account. The documentation of expenditures was irregular, and the account was inactive for up to 16 months, even though the disbursement letter stipulated that withdrawals should be made at least quarterly”. It was reported by the Project team that ratings for both financial management and procurement were satisfactory, however, there were some major shortcomings in the procurement system.

The ICR reported some contradicting information on procurement performance. As mentioned in Section 9 b, the ICR reported significant delays in project implementation that occurred because of procurement problems. However, then ICR also reported that (p.11) “the majority of contracts awarded according to schedule and a few suffering delays due to low capacity of the initial implementing agency”. The Bank recommended to the implementing agency to hire a qualified procurement specialist experienced in international procurement, this had positive results. A detailed Grant Recipient Handbook was prepared for implementation of the grant program. However, the physical inspection of grants under the project was inadequate. As a result of staff shortages, the Project Coordination Center did not visit grant recipients on a regular basis and the financial agent under-performed. This was only partly compensated by regular visits of the FAO implementation support team. Furthermore, as mentioned in Section 9b, due to the dissolving of the Project Coordination Center as of June 2009, the follow up of two ongoing activities could not be carried out and the Ministry of Agriculture considered all payments after June 30, 2012 as not legitimate and cancelled funding. However, these were contractional obligations with potential to damage the reputation of both the Bank and the government.

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Satisfactory
Moderately Unsatisfactory
The relevance of design is rated modest as the link between objectives and project activities and funding were not clear. Under efficacy the main objective of increasing the competitiveness of the agricultural sector is rated modest as there is not sufficient evidence to show that this was achieved. Efficiency is rated modest mainly due to weak economic analysis methodology. Overall, there were significant shortcomings. 
Risk to Development Outcome:
Moderate
Significant
The “competitiveness” objective is a risky objective, it is not only subject to macroeconomic conditions and additional measures on improved business environment and infrastructure, but also it is subject to what competitors do, which is outside national control.  
Bank Performance:
Moderately Satisfactory
Moderately Unsatisfactory
Both Quality at Entry and Quality of Supervision are rated Moderately Unsatisfactory, due to issues with results framework and technical design and during implementation the Bank team not being pro-active enough yielding the Moderately Unsatisfactory Rating.  
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:
This Review finds the following lessons suggested by the experience of this project:

1. Having a realistic development objective is key for project success. All elements of project evaluation are linked to the objectives- whether they are relevant, whether they are achieved and whether they are achieved efficiently. When a project has a very broad objective that is beyond the reasonable purview of the project, causal chain between inputs, outputs and outcomes/impacts are lost and achieving the objectives fully becomes very difficult.

2. Setting up and implementing a proper M&E system to measure the achievement of project is another key element for success. The M&E system should include all the related, measurable key outcome and intermediate indicators to adequately monitor and measure achievement of the objectives. The institutional arrangements as well as data collection/implementation methods and analysis should be clearly defined.

3. World Bank teams as well as the Government should be proactive and nimble in resolving project bottlenecks. When problems such as unrealistic objectives, inefficient M&E system, or poorly disbursing components become evident, Bank teams together with the Implementing Agency/Government should try to restructure/redesign the Project as early as possible to address the apparent issues. In particular, where possible government’s unduly bureaucratic procedures should be changed or adjusted for more efficient implementation.


14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The Implementation Completion Report was clear and well argued. There was frank description of challenges posed by the project design and of problems arising during implementation. Also there was detailed data on outputs and outcomes. But the ICR did not fully factor these into its performance ratings. The set of lessons drawn by the ICR could have assembled broader and more thoughtful insights out of the project experience. Finally, inconsistent reporting of total project and component costs through sometimes including beneficiaries’ contributions and at other times not. (eg US$83.1m (p. 3)) created confusion.


a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions