Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Modernizing Agricultural Knowledge & Information Systems Project (makis)


  
1. Project Data:   
ICR Review Date Posted:
06/25/2014   
Country:
Romania
PROJ ID:
P086949
Appraisal
Actual
Project Name:
Modernizing Agricultural Knowledge & Information Systems Project (makis)
Project Costs(US $M)
 71.98  51.99
L/C Number:
L4758
Loan/Credit (US $M)
 50.00  36.84
Sector Board:
Agriculture and Rural Development
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  11/16/2004
 
 
Closing Date
09/30/2010 03/25/2013
Sector(s):
Agricultural extension and research (75%), Central government administration (25%)
Theme(s):
Rural services and infrastructure (40% - P) Other rural development (20% - S) Rural policies and institutions (20% - S) Rural markets (20% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Ebru Karamete
George T. K. Pitman Christopher David Nelson IEGPS1

2. Project Objectives and Components:

a. Objectives:
The Project Appraisal Document (PAD, p.6) states that the project development objective is:


    "to assist the Romanian Government to improve the competitiveness of farmers and ago-processors in the EU accession environment".

The statement of project objectives in the Loan Agreement (p. 14) is:
    'to assist the Government to improve the competitiveness of farmers and agro-processors in the EU accession environment through: (i) better implementation of measures for inspection control, risk management and communication in food safety matters; (ii) strengthened capacity of the national research system to provide agricultural knowledge, skills, and information based on the needs of the agri-food sub-sector in line with EU requirements; (iii) improved capacity of research, extension, and food safety specialists to better serve agricultural producers’ needs in the context of the EU; and (iv) increased access of farmers and processors to knowledge of technologies related to production, quality control, food safety, processing and marketing in order to meet EU requirements."

Project development objectives were revised on September 30, 2010 as:
    "to help Romania comply with the agricultural acquis communautaire and to help the agro-food sector take advantage of the benefits and opportunities arising from EU membership".

This Review’s assessment is based upon the formulation of the project objective as in the Loan Agreement and the formally revised objectives.

b. Were the project objectives/key associated outcome targets revised during implementation?
Yes

If yes, did the Board approve the revised objectives/key associated outcome targets? Yes

Date of Board Approval: 09/30/2010

c. Components:
There were three components:

1. Strengthening the National Authority for Sanitary, Veterinary and Food Safety (ANSVSA) and the Phyto-sanitary Units (Appraisal Estimate: Euro 12.56 million, Revised: Euro EUR 6.8 million, Actual: Euro 4.63 million).
This component aimed to strengthen these institutions to: (i) implement the new food safety regulations adopted by Romania to comply with EU requirements; and (ii) provide the necessary control and verification of fresh and processed products to enable farmers, agro-industry and food processors to supply local and international markets. Activities planned to be funded included: institutional capacity building; laboratory needs under EU, Border Inspection Posts (BIPs); animal welfare; phytosanitary central laboratories; and a training program. However, with the exception of the completed BIPs (EUR 3.4 million), activities under Component 1 were cancelled in 2010.

2.Support for Agricultural Research (Appraisal Estimate: Euro 29.28 million, Actual: Euro 19.77 million).
This component aimed to strengthen the capacity of the national agricultural research system (NARS) to provide agricultural knowledge, skills and information based on the needs of the agri-food sub-sector consistent with EU requirements. This component included: (i) establishing National Reference Laboratories (NRLs) necessary to be functional immediately after accession for implementation of: nitrates and sludge directives, testing for Genetically Modified Organisms (GMO); (ii) a Reform Program to assist 5 Priority Research Institutes via technical assistance to develop reform strategies and an investment program; and (iii) expanding, and eventually mainstreaming, the ongoing Competitive Grant Program into Government agricultural research funding for core public sector activities.

3. Support for Advisory and Information Systems (Appraisal Estimate: Euro 9.56 million, Actual: Euro 11.05 million).
This component aimed to improve the capacity of the research, extension and food safety specialists to better serve the needs and improve competitiveness of farmers/ producers and processors in the context of EU accession; to increase access of farmers and processors to knowledge on technologies related to production, quality control, food safety, processing and marketing. There were two sub-components: (i) setting up a Training and Information Center (TIC); and (ii) improving effectiveness of advisory services. With project restructuring in 2010 one new activity was added under this component on support to improve the management information systems in agricultural administration through the Integrated Administration and Control System (IACS). This new activity aimed to ensure timely provision of EU agriculture support funds.

4. Project Management Unit (Appraisal Estimate: US$ Euro million, Actual: Euro 2.85 million).
This component funded for project co-ordination and administration, procurement, financial management, reporting, monitoring and evaluation (M&E) activities.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Costs:

Total costs were revised 3 times from the appraisal estimate of Euro 59.60 million to Euro 44.61 million (US$ 71.98 million, and US$ 51.99 million equivalent respectively). The actual total spending was Euro 38.30 million (64 % of the appraisal estimate). Changes included the reallocation of the uncommitted loan balance under Component 1 (EUR 6.8m) to Components 2 (EUR 6.15m) to cover increased costs of rehabilitation works of the National Research Institutes (NRIs) and the NRLs, and to Component 3 (EUR 0.65) for the IACS, to address the most pressing issues with regard to EU acquis compliance in the agro-food sector.

Financing: Loan amount estimate at appraisal was Euro 41.40 million (US$ 50.00 million). During the 3 project restructurings, a total of Euro 14.26 million Bank (US$ 18.75 million equivalent) was cancelled because of discontinuation of Component 1 and dropping of several activities under Component 2. At project closing, Euro 27.14 million (US$ 36.84 million equivalent) or 65.6 % of the original loan was disbursed.

Borrower Contribution:
It was estimated at appraisal that the Borrower contribution would be Euro 18.20 million (US$ 21.98 million equivalent), this amount was reduced to Euro 12.19 million with the 2010 restructuring and actual contribution was Euro 11.16 million (US$ 15.15 million equivalent) at closing (61 % of the original amount to be contributed by the Borrower).

Dates:
From the time of approval the project ran into numerous issues and delays and the project closing date of September 30, 2010 was extended for 30 months to March 25, 2013 in order to complete activities and contracts already procured. The project was officially restructured on September 30, 2010 due to changes in country priorities in post EU accession context as well as implementation issues mainly related to Component 1 whose original targets could not be achieved.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Original Project:
High

    The original project development objectives of improving the competitiveness of farmers and agro-processors in the EU accession environment were relevant to country priorities and strategies, and remained relevant through closing. Romania was scheduled for European Union (EU) accession in 2007. Despite agriculture’s dominant role in employment and land use, agricultural production had become increasingly inefficient and fragmented, with labor and land productivity only 6% and 27% of the respective averages in the EU. Agriculture employed 44% of Romania’s total labor force and 70% (4.8 million people) of its rural labor force, with the latter representing 72% of total agricultural labor in all the EU-15 countries. Agriculture also utilized 62% of Romania’s total area--one of the highest shares of cultivated land in all EU member states. Household farms dominated, with over 4.5 million agricultural holdings varying from less than 1 ha to 2,000 ha in size, and producing about 87% of agricultural output.

    Romania had made progress to harmonize laws and regulations to meet EU requirements but the government faced a major challenge in creating a competitive agricultural sector that was in compliance with EU requirements. Helping the farming community to meet relevant EU directives would allow them also to capture the benefits of EU accession. The project development objective was also consistent with the Government’s Strategy for the Development of Agriculture, Food Industry and Forestry for the medium- and long-term (2001-2005 and 2005-2010). Project objectives were relevant to the May 2009 Convergence Program of the Government of Romania, which has as its objective joining the Eurozone in 2014.

    The project development objective was relevant to the recent Country Partnership Strategy (FY2009-2013), specifically to the Growth and Competitiveness Pillar, which included the agriculture objective: "to provide advisory, technical and financial assistance to support the market-based restructuring and competitiveness of Romanian agriculture.”

Revised Project:
High
    The revised project development objectives with a focus of assisting compliance with the agricultural acquis communautaire were also highly relevant.

b. Relevance of Design:
Original Project: Substantial


    The project was a result of Government's request from Bank to fill specific gaps in its compliance efforts that were not being funded by the EU or other donors. Specifically the Ministry of Agriculture, Forests and Rural Development needed to strengthen its readiness to implement its newly designed policies in line with EU accession conditions. The earlier Bank project, Agricultural Support Services Project (ASSP) had highlighted the weaknesses of the existing research, extension and education systems, and project design focused on modernizing Agricultural Knowledge and Information System (AKIS) needed to meet the needs of farmers in the context of EU accession.

    The project design presented a logical causal chain between the project activities and the expected outcomes and was substantially relevant. In addition project design complemented EU activities with its focus on particular areas in food safety, research and extension. Agricultural research and extension advisory services, training of trainers as well as training of farmers on technologies including production, quality control, food safety, processing and marketing, were directly linked to achieving the competitiveness objective and were relevant.

    Revised Project: Substantial

    The new design improved the results chain by eliminating activities that could not deliver. The new design cancelled parts of Component 1 (except Border Inspection Points), and included management information system improvements in agricultural administration through the inclusion of a new component, the Integrated Administration and Control System (IACS) for EU Common Agricultural Policy Funds (CAP). This inclusion provided additional economic benefits to those identified at the appraisal stage. The investment assisted Romania in avoiding an annually recurring fine of approximately EUR 42 million that the EC might impose should Romania not comply with the requirements for the IACS system. Avoidance of such potential penalty would itself pay for the entire project within the project extension period. In addition, for every month the payment to farmers is delayed beyond the established deadline the EU’s reimbursement for the Government’s costs is reduced by 10%. It is also likely that the improved system will lower the administration costs for handling the CAP support, currently 12% of the total support. Also, refocusing farmer training on how to absorb EU funds was also relevant.


4. Achievement of Objectives (Efficacy) :

Original Project: “To improve the competitiveness of farmers and agro-processors in the EU accession environment”
Outcome (Original Objective): Modest

    Although project activities contributed to achieving the increased competitiveness objective, the ICR did not provide data on this. The project had 3 outcome indicators- agricultural productivity, agricultural income, and agricultural exports but these indicators were not monitored. Reason for not monitoring these outcome indicators ( ICR p. 10) were the difficulties on linking project activities to macro-level variables such as productivity and incomes that are influenced by many other factors, as well as existence of natural lags in observing impacts of changes in reform, training, and technology adoption. The ICR stated that (p. 13), farmers reported productivity increases of 25%, based on low starting values and a high degree of innovation and disseminated technologies (the adoption rate of technologies was reported as 49.8%). However, the methodology used to arrive at these numbers was not provided.

    In terms of intermediate outcomes, there is evidence on the improved capacity of national research institute as well as improved capacity of research, extension, and food safety specialists to service the information and technical needs of agricultural producers. Regional labs for soil testing and GMO labs that were equipped and accredited may also help towards achieving high food safety standards, but this was only partially achieved (see Outputs). Even so, there is no evidence related to production, quality control, food safety, processing and marketing that may have resulted from increased farmer and processor access to new knowledge and technologies.

    The activities related to food safety were cancelled at restructuring because of lack of progress and there is no evidence on better implementation of food safety matters. On December 3, 2009, the food safety responsibilities were transferred to the Food Safety Agency, which was set up as an independent agency under the Prime Minister’s office. This move made it a separate agency with the mandate to perform its professional directives to conduct food safety and border inspection independent of any ministry. However, the new management of this agency showed limited commitment to continue to implement the project activities (ICR p. 18). While Border Inspection Posts monitored internationally traded food products there is no evidence about how they had improved their effectiveness.



Revised Objective: “"to help Romania comply with the agricultural acquis communautaire and to help the agro-food sector take advantage of the benefits and opportunities arising from EU membership".

Outcome (Revised Objectives): Modest

    Gross agricultural value-added data for 2011 show that agri-food exports to EU 27 represented 9% of total Romanian exports to EU 27 (target 5.5%) relative to a baseline of 3.6%. While Romania has experienced improving export trends since 2007, it is difficult, however, to directly attribute this increase to the activities/outputs under the project. The ICR stated that there are several factors, including trade policy and exchange rates in recent years that have favored high levels of exports.

    The final impact assessment showed European Agricultural Guarantee Fund (EAGF) (committed/allocated) at 94% and European Agricultural Fund for Rural Development (EAFDR) (committed/allocated) at 50%, therefore the target was reached.

    The farmer training on EU CAP pillars 1 and 2 eligible activities for CAP funding, would in part have contributed to the increased commitment in CAP funding. The ICR did not provide these funds’ commitments. Also, the project investments helped to more easily track these commitments. However, as the ICR also mentioned, it is difficult to attribute this achievement solely to the project.


Outputs under the original Objectives:

Strengthening the National Agency for Sanitary, Veterinary and Food Safety (ANSVSA) and the Phyto-sanitary Unit:
  • 3 Border Inspection Posts (BIPs) in Otopeni, Constanta North and Constanta South were established and equipment was provided.
  • 2 Regional Laboratories of National Authority for Sanitary, Veterinary and Food Safety (ANSVSA) in Bucharest and Brasov were equipped. This was less than the target of 3 Regional Laboratories, as the procurement of the lab equipment for the Regional Lab in Prahova could not be not launched due to delays in renovation.
  • The target of 4 functional National Reference Laboratories could not be met.
  • Initially planned construction of the new building for the Institute for Veterinary Hygiene and Public Health (IVHPH) could not be carried out due to the lengthy process for the bidding process for the design and feasibility study for the building as well as a financing gap of EUR 7 million for this activity. There was no agreement also on potential financing sources to cover the costs associated with the design and construction of the IVHPH building.
  • As part of building a functional phyto-sanitary laboratory network, 2 National Reference Laboratories and 3 Regional Laboratories were equipped and became functional and staff trained (achieving the target)
  • 488 food safety inspectors were trained; and 32 ANSVSA staff were trained internationally.

Support for Agricultural Research:
  • Under the Competitive Grants Scheme (CGS), funds were made available to fund research institutions. In total, the CGS funded 44 Grant Agreements totaling Euro 7.68 million and 30 new technologies were developed. The main beneficiaries were Institutes/Research Stations (13 projects for 33% of total value), Universities (13 projects for 28% of total value, private companies (9 projects for 25% of total value) and NGOs (6 projects for 14% of total value).
  • Four priority research programs were selected for support by the project based on comparative advantage for support under the project. The project supported international consultants to develop work plans, research areas and training for researchers. The National Research Institute for Animal Biology and Nutrition (INBA) Balotesti, and the National Agricultural Research and Development Institute (NARDI) Fundulea were awarded an A+ rating during the evaluation process coordinated by the National Agency for Scientific Research. Through the reform process, NARDI improved the researchers to- auxiliary staff ratio from 1:3.3 (2005) to 1:1.9 (2013), and has created a research environment favorable for project-based teams and skills development (including through international grants). As a result, the research institute has increased the share of own revenues to 61% of total in 2012 (from 43% in 2006), has continued to strengthen its public-private partnerships, and has made significant inroads into the research of conservation and organic agriculture in Romania and crop diversification, its main research themes
  • The Research Institutes were able to develop new technologies, new varieties of plants, develop staff skills and enhance their networking (through training and study tours, most financed by the project) leveraging additional financial resources (i.e. European/international research grants). The institutes developed and tested over 30 new varieties of plants (cereals, potato, sugar beet, sunflower) more resistant to adverse climate conditions (drought, rainstorms, late freeze, etc.) and specific diseases; developed over 25 new technologies for bio-control of insects, and reduce use of pesticides and fertilizers; developed codes for good agricultural practices, technologies for remediation of lands polluted with heavy metals, oil and mining activities, and nutrient management plans in Nitrate vulnerable zones. All these achievements contributed to compliance with the Romanian and EU environment requirements. These technologies are being transferred to farmers through programs implemented by each institute.
  • The Genetically Modified Organisms (GMO) lab was equipped and accredited.
  • The originally planned procurement of laboratory equipment for the National Research and Development Institute for Soil Science, Agro-chemistry and Environment Protection (RISSA) Bucharest did not take place due to seismic risk of the lab building. The project dropped this activity during the 3rd restructuring, and cancelled EUR 3.5m. The original target indicator was not met, and it was revised to one NRL (for GMO testing).

Advisory and Information Systems:
  • Training and Information Centers in Bucharest, Timisoara and Cluj were fully functional. 1,900 extension consultants, 225 researchers, 400 food safety inspectors, and 5328 agro-processors/private sector were trained. The training was undertaken for trainers, specialists from the field of agricultural extension/consulting, food safety and research. Training modules emphasized areas that were specific to trainees’ agro-farming areas and lasted for 120 hours. Information/knowledge was also provided on new technologies from abroad according to the EU requirements.
  • 1.3 million farmers (against a target of 33,000) were reached and trained on EU accession, specifically on what activities are eligible for CAP funding under Pillars 1 and 2 of the EU program.
  • 11 information campaigns were carried out (against a target of 2).
  • Project funds initially allocated for procurement of software and technical assistance for improving the Integrated Administration and Control System (IACS) were reallocated to the procurement of IT hardware (server and database) for the Agency for Payments and Interventions for Agriculture (APIA). APIA staff completed the data migration from the old equipment to the new one, and received adequate training to utilize the new system.

Outputs under the Revised Objective.
  • Project activities including establishment of 3 Border Inspection Posts, and equipping the phyto-sanitary NRLs, the Regional Reference Laboratories and the GMO lab help towards achieving the high food safety standards of the EU. The CGS activity assisted in developing new technologies that are being disseminated to farmers to promote better farming practices and eventually raise productivity, and training of numerous beneficiaries, including farmers, food processors and food safety specialists should increase the overall efficiency of Romanian agriculture and the agri-food sector exports. The new technologies developed codes for good agricultural practices, technologies for remediation of lands polluted with heavy metals, oil and mining activities, and nutrient management plans in Nitrate vulnerable zones to contribute to compliance with the Romanian and EU environment requirements.
  • The CGS Impact Assessment of 2011 looked at 40 projects under the CGS. Out of 126 technologies disseminated to the target group, 89 were perceived as new by farmers. The main impacts mentioned by farmers were improvements in farming practices and production and quality increases.
  • The training modules developed and implemented under the Training Information Centers covered specific elements related to EU standards, new technologies related to production, quality control, food safety, and processing and marketing.
  • Hardware provided under the project to the Agency for Payments and Interventions in Agriculture (APIA) is the core element of implementing the new EU requirements regarding the Integrated Administration and Control System (IACS) for EU-CAP funds, to improve the processing of applications for area payments, enhance the management of farmers’ database and support the implementation of the agreed action plan towards improving the administration of EU rural funds. Installation of the system was done in December 2012 and became operational in early February 2013. It is unclear if this activity had time to contribute to the achievement of project outcomes.

5. Efficiency:


Economic and Financial Efficiency

    The economic assessment conducted at appraisal analyzed returns from 20 aggregated research and extension projects using data from the previous Agricultural Support Services Project. The aggregate economic rate of return (EIRR) amounted to 134 % and the benefit-cost ratio was estimated at 8.4. However these values could not be compared with the ex-post evaluation as the assumptions and the methodology is not known.

    The ICR did not present a separate efficiency analysis for the whole project for the original and revised project. Instead, the economic analysis was based on ex-post evaluation of 32 of the 41 Competitive Grant Projects, which comprised approximately 19 % of total project costs. The projects included the crops, livestock and natural resources sectors. For the other 9 projects, the evaluation could not be done either because they were focused on marketing, management or information systems and therefore it was not possible the quantify costs and benefits or they did not provide accurate data

    The ICR (p. 33) provided an example of ERR/NPV calculation (duration 15 years, discount rate 12 %) for an extension project on agricultural technologies of varieties of straw, cereals, canola, sunflower and maize carried out by a research institution. The aggregate ERR calculated under these assumptions was 88 %. There are questions about the validity of the analysis:

        (i) Although the project included technologies for several crops, the calculations included canola only, it was not explained why only canola was selected.
        (ii) It was not clear how the gross margins (incremental benefits) were calculated, and what the assumptions were for calculating the project and non-project yields, operational costs and revenues.
        (iii) Furthermore, the ERR calculation compared project beneficiary village's size of canola fields with the non-project village's canola fields to come up with the incremental canola areas, and the incremental benefit per ha was assumed constant. The assumption was that the canola areas increased with the help of the project, but with a decreasing trend each year. However this method ignored the fact that as the canola areas in with-project village increased, the without-project village did not probably leave its fields idle, but cultivated other crops and therefore had benefits coming from those other crops. Ignoring this additional benefit coming from other crops, probably led to overestimating the total incremental benefits.
        (iv) The example project yielded an ERR of 98 % but the ICR did not provide ERR results for the remaining 31 projects.

Administrative and Institutional Efficiency
    The project had a number of administrative and operational inefficiencies. Frequent changes in government and Ministers of Agriculture and the insufficient focus on priorities in Ministry of Agriculture and Rural Development negatively affected overall project implementation. The continuous non-performance of The National Veterinary and Food safety Authority and the lack of any concrete plan to remedy this situation led to the decision to discontinue all the activities, and led to the cancellation of important activities in food safety. Furthermore, the project suffered occasionally from insufficient allocation of government funds and in many instances from delayed allocation of funds for the project activities, which substantially affected implementation throughout the project’s life. Unfilled vacancies within the PMU affected some of the PMU responsibilities and project implementation (i.e. hiring of M&E expert). As a result of these problems, the project closing date was extended for approximately 2.5 years from September 30 2010 to March 25, 2013. In addition, these project delays and inefficiencies led to the cancellation of 34% of the project loan.

Efficiency of the project is rated modest

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
134%
100%
ICR estimate:
Yes
88%
19%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Original Project Outcome:
Relevance of objectives is high, and relevance of design is substantial. Efficacy is rated modest due to lack of evidence on achieving the objectives. Efficiency is rated modest mainly due to weakness of the economic analysis and operational inefficiencies. These are significant shortcomings in the achievement of the project’s efficacy and efficiency and its outcome is rated Moderately Unsatisfactory.


Revised Project Outcome:
Relevance of objectives is high, and relevance of revised design is substantial. Efficacy is rated modest primarily because of attribution issues and the lack of a counterfactual. Efficiency cannot be rated separately and the overall rating of modest is retained. These are significant shortcomings in the achievement of the project’s design, efficacy and efficiency and its outcome is rated Moderately Unsatisfactory.

Overall Project Outcome:
The combined outcome rating of a restructured project is weighted according to the proportion of the Credit that was disbursed before and after the restructuring. However, as outcome rating is the same for both phases of the project, the overall outcome is rated moderately unsatisfactory

a. Outcome Rating: Moderately Unsatisfactory

7. Rationale for Risk to Development Outcome Rating:

The main risks are institutional, economic and financial. In terms of institutional risk, while the reform of the research institutes represents a critical achievement, the Government will have to provide institutes adequate funding so that these institutions can continue to focus on core research activities and be able deliver the needed technologies to improve competitiveness of the farmers. To achieve this a significant increase in public funding for agricultural research and extension is essential. The government has recently become committed in reforming research activities; i.e. an indication on Government determination to perform a real restructuring of the research sector is the recent governmental decision placing all the national research and development institutes under the coordination of the Ministry of National Education. Furthermore, the research and innovation funding has started slowly to grow, reaching in 2012 a similar level to that in 2007.


    The Government also has committed to develop a comprehensive action plan aimed at modernizing the public administration, including strengthening MADR management and operational capacity, through a World Bank technical assistance project, which will allow for a better absorption of EU funds. In addition, a number of sub-project types financed from the project CGS would continue to be financed from the CAP-Pillar 2. The new provisions of the CAP promoting the partnership for innovation (among farmers, research units, public administration) would also bring new opportunities for these institutes to align their activities with the demands of farmers.

    a. Risk to Development Outcome Rating: Moderate

8. Assessment of Bank Performance:

a. Quality at entry:
The project preparation team worked closely with the EU, as well as several EU member countries to draw experiences to ensure that the Project covered the critical gaps in implementation capacity. The team reviewed compliance efforts of EU member countries and consulted with a broad spectrum of stakeholders, including small, medium and large-scale farmers, agro-processors, producer groups, farmers’ associations, women groups, and NGOs. Project design reflected the following lessons: (i) including both public and private sector services in designing research and extension systems; (ii) ensuring local ownership and involving farmers in determining research agendas and priorities; (iii) enhancing research quality through needs-based training, competitive research grants, and by involving universities and a range of research providers.
The Project introduced mechanisms to mitigate the effects of potential risks. In order to mitigate the risk of resistance to change coming from within the research community, the project focused on four priority research institutes that would also receive core funding from the Government. Procurement and financial management was adequately appraised. However, institutional arrangements were not properly designed. Selection of ANSVSA as one implementing agency proved to be problematic: a more thorough assessment of its institutional capacity, including provisions for capacity building and/or means to attract institutional commitment on implementation, was needed. As result of this shortcoming, ANSVSA was dropped from the project and this led to cancellation of significant share of project financing.

The Results Framework had significant shortcomings also in terms delinked outcome indicators. An essential baseline survey was not conducted (See section 10).

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

Supervision was regular and there were three Task Team leaders. Project Aide Memoires and ISRs were descriptive, failed to report or mention any progress towards the desired outcomes, and were not always realistic. The team could have acted upon revising the results framework much earlier, as it was clear that outcome indicators were not adequate and were not being monitored

The Bank remained engaged with the client throughout. However, frequent changes in political leadership, executive staff and the removal of key reformers negatively affected project implementation. Additionally, insufficient government budget allocation to the project delayed implementation and led to cancellation of some activities and extension of the project by 30 months – all factors beyond the Bank’s control. Eventually, and after considerable delay, the project was restructured in 2010 to address mounting implementation problems and the inadequate results framework. While this effectively turned the project around it was too late to significantly change the outcome, particularly as even the revised results framework had some problems.

The Bank was proactive in identifying issues related to procurement, especially with respect to Component 1 earlier on in the project, and also in recognizing and requesting correction of the updated procurement plan in November 2012. Attention to financial management and safeguards was satisfactory.

Quality of Supervision Rating: Moderately Unsatisfactory

Overall Bank Performance Rating: Moderately Unsatisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The government engaged with the project closely during preparation. Also it made the necessary reforms to the agricultural research system, which should have long-lasting effects on Romania's research capabilities, including the enhanced ability to attract and train researchers, and improving the dissemination of technologies to farmers. However, the government failed to allocate the resources agreed on at appraisal (equivalent of EUR 2.0 million in the 2005 budget), thus delaying the start of project activities for approximately 1 year. Insufficient and/or delayed funding for the project continued and led to further delays and uncertainties. As a result, some activities were cancelled due to inadequate time for completion. Furthermore, frequent changes in leadership, lack of timely decision-making, and lack of commitment, especially on major food safety activities, led to their cancellation, and negatively affected overall project achievements. In addition the financial and Euro zone crises were unexpected factors that further complicated the budget allocation issue.

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:

The project had two implementing agencies. The project management unit (PMU) in ANSVSA, the agency responsible for food safety was designated to implement Component 1, while the PMU in MAFRD was responsible for the other two components.

ANSVSA showed major shortcomings in implementation in terms of continuous nonperformance and weak capacity to provide institutional leadership. Despite discussions with the Bank on alternative initiatives, ANSVSA did not pursue the agreed steps and completely lacked any concrete plan to remedy the situation. This led to a restructuring to discontinue the activities coordinated by ANSVSA and the termination of ANSVSA’s implementation responsibilities. The food safety responsibilities were transferred to the Food Safety Agency, which was set up as an independent agency under the Prime Minister’s office. However, the new management of this agency also showed limited commitment to implementing project activities.

The PMU of MAFRD performed the financial and accounting functions adequately, having also become familiar with Bank procedures during the implementation of the previous Bank project, ASSP. With regard to M&E, the PMU did not conduct a baseline survey as planned at project design, and overall project M&E was weak, an issue exacerbated by the absence of an M&E Specialist in the PMU for long periods of time. There has been no monitoring of the Project’s original or revised outcome indicators, and the Impact Assessment, which was discussed in 2010, was conducted only in February/March 2013 and completed in June 2013.

Procurement was problematic with several deficiencies, i.e. delays in signing contracts in ANSVSA beyond acceptable standards, the necessity to re-bid several contracts, and splitting of bigger packages into smaller packages. The PMU’s difficulties in hiring procurement staff further added to these problems, resulting in slow implementation during several months and as late as 2012, when the Bank advised on hiring additional procurement staff to prevent longer delays. Financial Management and Environmental Safeguards arrangements remained satisfactory throughout implementation.

Implementing Agency Performance Rating: Moderately Unsatisfactory

Overall Borrower Performance Rating: Moderately Unsatisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:
The original outcome indicators of 'increased adoption of improved farm practices, marketing, post-harvest and food safety technologies compatible with EU standards' and 'improved agricultural productivity' were linked to the original development objectives of improving competitiveness of farmers. Although the first outcome indicator was rather broad and difficult to measure, the second indicator was more straightforward. However, the inability to measure and attribute these indicators to the project necessitated dropping them at restructuring (ICR p.8). The intermediate outcome indicators on the other hand were more adequate, linked and measurable. Although agreed on at appraisal, the project did not conduct a baseline survey to use as a benchmark against which to measure progress and impact.

b. M&E Implementation:
At restructuring in 2010 the new outcome indicators included 'Increase in agri-food exports to EU relative to Gross Value Added' and "Increase in CAP funding committed to beneficiaries under EU funds'. However the new outcome indicators were also problematic as they were not very well linked to the new development objectives (complying with agricultural acquis communautaire and helping the agro-food sector to take advantage of the benefits and opportunities arising from EU membership) and also were prone to be affected by many macro and micro factors outside the scope of the project.

There was no reporting on the progress towards the desired outcomes or a system set up to monitor the development indicators The PMU and the teams tracked regularly the progress achieved by the intermediate outcome indicators, but not the outcome indicators that were “impossible to measure and insufficiently related to the project development objectives” (ICR p. 8).

a. M&E Utilization:
The ICR did not report on M&E utilization.

M&E Quality Rating: Negligible

11. Other Issues:

a. Safeguards:

The project was classified as an environmental category ‘B’ under OP/BP4.01 Environmental Assessment as there were no significant long-term irreversible negative environmental impacts. An Environmental Assessment and an Environmental Management Plan were developed. In addition, Romania has a very inclusive national legislation that fully met the EU directives with regard to environmental protection and health and food safety measures. The potential impacts of the civil works were recognized as local in nature and mitigating measures were addressed through the permitting and construction process. The main potential adverse impact arose from the use and handling of hazardous chemical substances in laboratories and on-farms. In March 2010, the project faced issues about land ownership for the new RISSA building and the first batch of laboratory equipment procured under the project for the nitrates and sludge testing was installed in the main building of the institute instead. The procurement of the second batch of equipment (for soil testing) was cancelled because the laboratory building had structural weaknesses which could not be addressed by the project.

The Project’s Environmental Safeguards were rated satisfactory throughout implementation. In October 2012, the Bank mission concluded that all works were implemented in accordance with health and safety requirements and in compliance with the Bank’s environmental safeguards (ICR p. 8).

b. Fiduciary Compliance:

Financial Management arrangements remained satisfactory throughout implementation (ICR p. 9). Even so, the project suffered from the poor performance of the part-time procurement staff and financial management staff which resulted in delays in contracting and unclear/unreliable financial evidence.

Procurement was downgraded to ‘MS’ in 2007, and to ‘MU’ in June 2008 due to the ‘U’ rating for procurement in ANSVSA and several deficiencies, i.e. delays in signing contracts in ANSVSA beyond acceptable standards, the necessity to re-bid several contracts, and splitting of bigger packages into smaller packages. These also contributed to low overall disbursements. Processing of procurement packages continued to be delayed through 2010 due to ministry and government approval procedure. The PMU’s difficulties in hiring procurement staff further added to these problems, resulting in slow implementation during several months of the project and as late as 2012, when the Bank advised on hiring additional procurement staff to prevent longer delays. A post-procurement review identified some minor issues but found contracts under this area to be in compliance (ICR p.8).

c. Unintended Impacts (positive or negative):

d. Other:



12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Risk to Development Outcome:
Moderate
Moderate
 
Bank Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Borrower Performance:
Moderately Unsatisfactory
Moderately Unsatisfactory
 
Quality of ICR:
 
Satisfactory
 
NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

The ICR offers a number of lessons of which the following are the most important (with some reformulation of language):
    Strong attention to project implementation readiness is critical and may imply allowing for a longer project preparation period. This includes a careful assessment of institutional capacity and where necessary including provisions for capacity-building. If project activities include civil works, the feasibility studies and engineering designs should be available at appraisal, so that the works contracts can be procured as early as possible in the implementation calendar. Also, the project M&E system should be in place from the beginning of project implementation including indicators directly related to the project objectives.
    Strong linkages between the agricultural research and extension system are critical for the broad dissemination of technologies developed. The expected beneficiaries of the project results need to be identified and the tangible results from practical implementation of research outputs properly monitored.
    The stability of the project implementing unit staff, particularly the procurement and financial management specialists, is critical for timely and quality results in project execution. The project suffered of the poor performance of the part-time procurement staff and financial management staff which resulted in delays in contracting and unclear/unreliable financial evidence.
IEG also finds that:
    Having a realistic development objective is key for project success. When a project has a very broad objective that is beyond the reasonable purview of the project, the causal chain between inputs, outputs and outcomes/impacts is lost and achieving the objectives fully becomes very difficult.

14. Assessment Recommended?

Yes
Why?
Projects that are linked with the EU accession program are few and have special needs. A couple of similar projects were developed in ECA region over the same period of time. It would be valuable to take a close look at these projects to understand in more detail how this type of product can work elsewhere.

15. Comments on Quality of ICR:

The ICR was in many aspects well written, with a good narration of implementation progress and challenges. It is internally consistent and the conclusion follow from the evidence presented. However, the following points needed attention: (i) The ICR could have provided more information on the details and assumptions of the economic analysis; (ii) It could have provided more evidence to support some lessons drawn.

a. Quality of ICR Rating: Satisfactory

(ICRR-Rev6INV-Jun-2011)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions