|1. Project Data:
ICR Review Date Posted:
|Modernizing Agricultural Knowledge & Information Systems Project (makis)
Project Costs(US $M)
Loan/Credit (US $M)
|Agriculture and Rural Development
Cofinancing (US $M)
Board Approval Date
|Agricultural extension and research (75%), Central government administration (25%)|
|Rural services and infrastructure (40% - P)
Other rural development (20% - S)
Rural policies and institutions (20% - S)
Rural markets (20% - S)|
||ICR Review Coordinator:
||George T. K. Pitman
||Christopher David Nelson
|2. Project Objectives and Components:|
a. Objectives:The Project Appraisal Document (PAD, p.6) states that the project development objective is:
"to assist the Romanian Government to improve the competitiveness of farmers and ago-processors in the EU accession environment".
The statement of project objectives in the Loan Agreement (p. 14) is:
'to assist the Government to improve the competitiveness of farmers and agro-processors in the EU accession environment through: (i) better implementation of measures for inspection control, risk management and communication in food safety matters; (ii) strengthened capacity of the national research system to provide agricultural knowledge, skills, and information based on the needs of the agri-food sub-sector in line with EU requirements; (iii) improved capacity of research, extension, and food safety specialists to better serve agricultural producers’ needs in the context of the EU; and (iv) increased access of farmers and processors to knowledge of technologies related to production, quality control, food safety, processing and marketing in order to meet EU requirements."
Project development objectives were revised on September 30, 2010 as:
"to help Romania comply with the agricultural acquis communautaire and to help the agro-food sector take advantage of the benefits and opportunities arising from EU membership".
This Review’s assessment is based upon the formulation of the project objective as in the Loan Agreement and the formally revised objectives.
b. Were the project objectives/key associated outcome targets revised during implementation?
If yes, did the Board approve the revised objectives/key associated outcome targets?
Date of Board Approval: 09/30/2010
c. Components:There were three components:
1. Strengthening the National Authority for Sanitary, Veterinary and Food Safety (ANSVSA) and the Phyto-sanitary Units (Appraisal Estimate: Euro 12.56 million, Revised: Euro EUR 6.8 million, Actual: Euro 4.63 million).
This component aimed to strengthen these institutions to: (i) implement the new food safety regulations adopted by Romania to comply with EU requirements; and (ii) provide the necessary control and verification of fresh and processed products to enable farmers, agro-industry and food processors to supply local and international markets. Activities planned to be funded included: institutional capacity building; laboratory needs under EU, Border Inspection Posts (BIPs); animal welfare; phytosanitary central laboratories; and a training program. However, with the exception of the completed BIPs (EUR 3.4 million), activities under Component 1 were cancelled in 2010.
2.Support for Agricultural Research (Appraisal Estimate: Euro 29.28 million, Actual: Euro 19.77 million).
This component aimed to strengthen the capacity of the national agricultural research system (NARS) to provide agricultural knowledge, skills and information based on the needs of the agri-food sub-sector consistent with EU requirements. This component included: (i) establishing National Reference Laboratories (NRLs) necessary to be functional immediately after accession for implementation of: nitrates and sludge directives, testing for Genetically Modified Organisms (GMO); (ii) a Reform Program to assist 5 Priority Research Institutes via technical assistance to develop reform strategies and an investment program; and (iii) expanding, and eventually mainstreaming, the ongoing Competitive Grant Program into Government agricultural research funding for core public sector activities.
3. Support for Advisory and Information Systems (Appraisal Estimate: Euro 9.56 million, Actual: Euro 11.05 million).
This component aimed to improve the capacity of the research, extension and food safety specialists to better serve the needs and improve competitiveness of farmers/ producers and processors in the context of EU accession; to increase access of farmers and processors to knowledge on technologies related to production, quality control, food safety, processing and marketing. There were two sub-components: (i) setting up a Training and Information Center (TIC); and (ii) improving effectiveness of advisory services. With project restructuring in 2010 one new activity was added under this component on support to improve the management information systems in agricultural administration through the Integrated Administration and Control System (IACS). This new activity aimed to ensure timely provision of EU agriculture support funds.
4. Project Management Unit (Appraisal Estimate: US$ Euro million, Actual: Euro 2.85 million).
This component funded for project co-ordination and administration, procurement, financial management, reporting, monitoring and evaluation (M&E) activities.
d. Comments on Project Cost, Financing, Borrower Contribution, and DatesProject Costs:
Total costs were revised 3 times from the appraisal estimate of Euro 59.60 million to Euro 44.61 million (US$ 71.98 million, and US$ 51.99 million equivalent respectively). The actual total spending was Euro 38.30 million (64 % of the appraisal estimate). Changes included the reallocation of the uncommitted loan balance under Component 1 (EUR 6.8m) to Components 2 (EUR 6.15m) to cover increased costs of rehabilitation works of the National Research Institutes (NRIs) and the NRLs, and to Component 3 (EUR 0.65) for the IACS, to address the most pressing issues with regard to EU acquis compliance in the agro-food sector.
Financing: Loan amount estimate at appraisal was Euro 41.40 million (US$ 50.00 million). During the 3 project restructurings, a total of Euro 14.26 million Bank (US$ 18.75 million equivalent) was cancelled because of discontinuation of Component 1 and dropping of several activities under Component 2. At project closing, Euro 27.14 million (US$ 36.84 million equivalent) or 65.6 % of the original loan was disbursed.
It was estimated at appraisal that the Borrower contribution would be Euro 18.20 million (US$ 21.98 million equivalent), this amount was reduced to Euro 12.19 million with the 2010 restructuring and actual contribution was Euro 11.16 million (US$ 15.15 million equivalent) at closing (61 % of the original amount to be contributed by the Borrower).
From the time of approval the project ran into numerous issues and delays and the project closing date of September 30, 2010 was extended for 30 months to March 25, 2013 in order to complete activities and contracts already procured. The project was officially restructured on September 30, 2010 due to changes in country priorities in post EU accession context as well as implementation issues mainly related to Component 1 whose original targets could not be achieved.
|3. Relevance of Objectives & Design:|
a. Relevance of Objectives:Original Project:
The original project development objectives of improving the competitiveness of farmers and agro-processors in the EU accession environment were relevant to country priorities and strategies, and remained relevant through closing. Romania was scheduled for European Union (EU) accession in 2007. Despite agriculture’s dominant role in employment and land use, agricultural production had become increasingly inefficient and fragmented, with labor and land productivity only 6% and 27% of the respective averages in the EU. Agriculture employed 44% of Romania’s total labor force and 70% (4.8 million people) of its rural labor force, with the latter representing 72% of total agricultural labor in all the EU-15 countries. Agriculture also utilized 62% of Romania’s total area--one of the highest shares of cultivated land in all EU member states. Household farms dominated, with over 4.5 million agricultural holdings varying from less than 1 ha to 2,000 ha in size, and producing about 87% of agricultural output.
Romania had made progress to harmonize laws and regulations to meet EU requirements but the government faced a major challenge in creating a competitive agricultural sector that was in compliance with EU requirements. Helping the farming community to meet relevant EU directives would allow them also to capture the benefits of EU accession. The project development objective was also consistent with the Government’s Strategy for the Development of Agriculture, Food Industry and Forestry for the medium- and long-term (2001-2005 and 2005-2010). Project objectives were relevant to the May 2009 Convergence Program of the Government of Romania, which has as its objective joining the Eurozone in 2014.
The project development objective was relevant to the recent Country Partnership Strategy (FY2009-2013), specifically to the Growth and Competitiveness Pillar, which included the agriculture objective: "to provide advisory, technical and financial assistance to support the market-based restructuring and competitiveness of Romanian agriculture.”
The revised project development objectives with a focus of assisting compliance with the agricultural acquis communautaire were also highly relevant.
b. Relevance of Design:Original Project: Substantial
The project was a result of Government's request from Bank to fill specific gaps in its compliance efforts that were not being funded by the EU or other donors. Specifically the Ministry of Agriculture, Forests and Rural Development needed to strengthen its readiness to implement its newly designed policies in line with EU accession conditions. The earlier Bank project, Agricultural Support Services Project (ASSP) had highlighted the weaknesses of the existing research, extension and education systems, and project design focused on modernizing Agricultural Knowledge and Information System (AKIS) needed to meet the needs of farmers in the context of EU accession.
The project design presented a logical causal chain between the project activities and the expected outcomes and was substantially relevant. In addition project design complemented EU activities with its focus on particular areas in food safety, research and extension. Agricultural research and extension advisory services, training of trainers as well as training of farmers on technologies including production, quality control, food safety, processing and marketing, were directly linked to achieving the competitiveness objective and were relevant.
Revised Project: Substantial
The new design improved the results chain by eliminating activities that could not deliver. The new design cancelled parts of Component 1 (except Border Inspection Points), and included management information system improvements in agricultural administration through the inclusion of a new component, the Integrated Administration and Control System (IACS) for EU Common Agricultural Policy Funds (CAP). This inclusion provided additional economic benefits to those identified at the appraisal stage. The investment assisted Romania in avoiding an annually recurring fine of approximately EUR 42 million that the EC might impose should Romania not comply with the requirements for the IACS system. Avoidance of such potential penalty would itself pay for the entire project within the project extension period. In addition, for every month the payment to farmers is delayed beyond the established deadline the EU’s reimbursement for the Government’s costs is reduced by 10%. It is also likely that the improved system will lower the administration costs for handling the CAP support, currently 12% of the total support. Also, refocusing farmer training on how to absorb EU funds was also relevant.
|4. Achievement of Objectives (Efficacy) :|
Original Project: “To improve the competitiveness of farmers and agro-processors in the EU accession environment”
Outcome (Original Objective): Modest
Although project activities contributed to achieving the increased competitiveness objective, the ICR did not provide data on this. The project had 3 outcome indicators- agricultural productivity, agricultural income, and agricultural exports but these indicators were not monitored. Reason for not monitoring these outcome indicators ( ICR p. 10) were the difficulties on linking project activities to macro-level variables such as productivity and incomes that are influenced by many other factors, as well as existence of natural lags in observing impacts of changes in reform, training, and technology adoption. The ICR stated that (p. 13), farmers reported productivity increases of 25%, based on low starting values and a high degree of innovation and disseminated technologies (the adoption rate of technologies was reported as 49.8%). However, the methodology used to arrive at these numbers was not provided.
In terms of intermediate outcomes, there is evidence on the improved capacity of national research institute as well as improved capacity of research, extension, and food safety specialists to service the information and technical needs of agricultural producers. Regional labs for soil testing and GMO labs that were equipped and accredited may also help towards achieving high food safety standards, but this was only partially achieved (see Outputs). Even so, there is no evidence related to production, quality control, food safety, processing and marketing that may have resulted from increased farmer and processor access to new knowledge and technologies.
The activities related to food safety were cancelled at restructuring because of lack of progress and there is no evidence on better implementation of food safety matters. On December 3, 2009, the food safety responsibilities were transferred to the Food Safety Agency, which was set up as an independent agency under the Prime Minister’s office. This move made it a separate agency with the mandate to perform its professional directives to conduct food safety and border inspection independent of any ministry. However, the new management of this agency showed limited commitment to continue to implement the project activities (ICR p. 18). While Border Inspection Posts monitored internationally traded food products there is no evidence about how they had improved their effectiveness.
Revised Objective: “"to help Romania comply with the agricultural acquis communautaire and to help the agro-food sector take advantage of the benefits and opportunities arising from EU membership".
Outcome (Revised Objectives): Modest
Gross agricultural value-added data for 2011 show that agri-food exports to EU 27 represented 9% of total Romanian exports to EU 27 (target 5.5%) relative to a baseline of 3.6%. While Romania has experienced improving export trends since 2007, it is difficult, however, to directly attribute this increase to the activities/outputs under the project. The ICR stated that there are several factors, including trade policy and exchange rates in recent years that have favored high levels of exports.
The final impact assessment showed European Agricultural Guarantee Fund (EAGF) (committed/allocated) at 94% and European Agricultural Fund for Rural Development (EAFDR) (committed/allocated) at 50%, therefore the target was reached.
The farmer training on EU CAP pillars 1 and 2 eligible activities for CAP funding, would in part have contributed to the increased commitment in CAP funding. The ICR did not provide these funds’ commitments. Also, the project investments helped to more easily track these commitments. However, as the ICR also mentioned, it is difficult to attribute this achievement solely to the project.
Outputs under the original Objectives:
Strengthening the National Agency for Sanitary, Veterinary and Food Safety (ANSVSA) and the Phyto-sanitary Unit:
- 3 Border Inspection Posts (BIPs) in Otopeni, Constanta North and Constanta South were established and equipment was provided.
- 2 Regional Laboratories of National Authority for Sanitary, Veterinary and Food Safety (ANSVSA) in Bucharest and Brasov were equipped. This was less than the target of 3 Regional Laboratories, as the procurement of the lab equipment for the Regional Lab in Prahova could not be not launched due to delays in renovation.
- The target of 4 functional National Reference Laboratories could not be met.
- Initially planned construction of the new building for the Institute for Veterinary Hygiene and Public Health (IVHPH) could not be carried out due to the lengthy process for the bidding process for the design and feasibility study for the building as well as a financing gap of EUR 7 million for this activity. There was no agreement also on potential financing sources to cover the costs associated with the design and construction of the IVHPH building.
- As part of building a functional phyto-sanitary laboratory network, 2 National Reference Laboratories and 3 Regional Laboratories were equipped and became functional and staff trained (achieving the target)
- 488 food safety inspectors were trained; and 32 ANSVSA staff were trained internationally.
Support for Agricultural Research:
- Under the Competitive Grants Scheme (CGS), funds were made available to fund research institutions. In total, the CGS funded 44 Grant Agreements totaling Euro 7.68 million and 30 new technologies were developed. The main beneficiaries were Institutes/Research Stations (13 projects for 33% of total value), Universities (13 projects for 28% of total value, private companies (9 projects for 25% of total value) and NGOs (6 projects for 14% of total value).
- Four priority research programs were selected for support by the project based on comparative advantage for support under the project. The project supported international consultants to develop work plans, research areas and training for researchers. The National Research Institute for Animal Biology and Nutrition (INBA) Balotesti, and the National Agricultural Research and Development Institute (NARDI) Fundulea were awarded an A+ rating during the evaluation process coordinated by the National Agency for Scientific Research. Through the reform process, NARDI improved the researchers to- auxiliary staff ratio from 1:3.3 (2005) to 1:1.9 (2013), and has created a research environment favorable for project-based teams and skills development (including through international grants). As a result, the research institute has increased the share of own revenues to 61% of total in 2012 (from 43% in 2006), has continued to strengthen its public-private partnerships, and has made significant inroads into the research of conservation and organic agriculture in Romania and crop diversification, its main research themes
- The Research Institutes were able to develop new technologies, new varieties of plants, develop staff skills and enhance their networking (through training and study tours, most financed by the project) leveraging additional financial resources (i.e. European/international research grants). The institutes developed and tested over 30 new varieties of plants (cereals, potato, sugar beet, sunflower) more resistant to adverse climate conditions (drought, rainstorms, late freeze, etc.) and specific diseases; developed over 25 new technologies for bio-control of insects, and reduce use of pesticides and fertilizers; developed codes for good agricultural practices, technologies for remediation of lands polluted with heavy metals, oil and mining activities, and nutrient management plans in Nitrate vulnerable zones. All these achievements contributed to compliance with the Romanian and EU environment requirements. These technologies are being transferred to farmers through programs implemented by each institute.
- The Genetically Modified Organisms (GMO) lab was equipped and accredited.
- The originally planned procurement of laboratory equipment for the National Research and Development Institute for Soil Science, Agro-chemistry and Environment Protection (RISSA) Bucharest did not take place due to seismic risk of the lab building. The project dropped this activity during the 3rd restructuring, and cancelled EUR 3.5m. The original target indicator was not met, and it was revised to one NRL (for GMO testing).
Advisory and Information Systems:
- Training and Information Centers in Bucharest, Timisoara and Cluj were fully functional. 1,900 extension consultants, 225 researchers, 400 food safety inspectors, and 5328 agro-processors/private sector were trained. The training was undertaken for trainers, specialists from the field of agricultural extension/consulting, food safety and research. Training modules emphasized areas that were specific to trainees’ agro-farming areas and lasted for 120 hours. Information/knowledge was also provided on new technologies from abroad according to the EU requirements.
- 1.3 million farmers (against a target of 33,000) were reached and trained on EU accession, specifically on what activities are eligible for CAP funding under Pillars 1 and 2 of the EU program.
- 11 information campaigns were carried out (against a target of 2).
- Project funds initially allocated for procurement of software and technical assistance for improving the Integrated Administration and Control System (IACS) were reallocated to the procurement of IT hardware (server and database) for the Agency for Payments and Interventions for Agriculture (APIA). APIA staff completed the data migration from the old equipment to the new one, and received adequate training to utilize the new system.
Outputs under the Revised Objective.
- Project activities including establishment of 3 Border Inspection Posts, and equipping the phyto-sanitary NRLs, the Regional Reference Laboratories and the GMO lab help towards achieving the high food safety standards of the EU. The CGS activity assisted in developing new technologies that are being disseminated to farmers to promote better farming practices and eventually raise productivity, and training of numerous beneficiaries, including farmers, food processors and food safety specialists should increase the overall efficiency of Romanian agriculture and the agri-food sector exports. The new technologies developed codes for good agricultural practices, technologies for remediation of lands polluted with heavy metals, oil and mining activities, and nutrient management plans in Nitrate vulnerable zones to contribute to compliance with the Romanian and EU environment requirements.
- The CGS Impact Assessment of 2011 looked at 40 projects under the CGS. Out of 126 technologies disseminated to the target group, 89 were perceived as new by farmers. The main impacts mentioned by farmers were improvements in farming practices and production and quality increases.
- The training modules developed and implemented under the Training Information Centers covered specific elements related to EU standards, new technologies related to production, quality control, food safety, and processing and marketing.
- Hardware provided under the project to the Agency for Payments and Interventions in Agriculture (APIA) is the core element of implementing the new EU requirements regarding the Integrated Administration and Control System (IACS) for EU-CAP funds, to improve the processing of applications for area payments, enhance the management of farmers’ database and support the implementation of the agreed action plan towards improving the administration of EU rural funds. Installation of the system was done in December 2012 and became operational in early February 2013. It is unclear if this activity had time to contribute to the achievement of project outcomes.
Economic and Financial Efficiency
The economic assessment conducted at appraisal analyzed returns from 20 aggregated research and extension projects using data from the previous Agricultural Support Services Project. The aggregate economic rate of return (EIRR) amounted to 134 % and the benefit-cost ratio was estimated at 8.4. However these values could not be compared with the ex-post evaluation as the assumptions and the methodology is not known.
The ICR did not present a separate efficiency analysis for the whole project for the original and revised project. Instead, the economic analysis was based on ex-post evaluation of 32 of the 41 Competitive Grant Projects, which comprised approximately 19 % of total project costs. The projects included the crops, livestock and natural resources sectors. For the other 9 projects, the evaluation could not be done either because they were focused on marketing, management or information systems and therefore it was not possible the quantify costs and benefits or they did not provide accurate data
The ICR (p. 33) provided an example of ERR/NPV calculation (duration 15 years, discount rate 12 %) for an extension project on agricultural technologies of varieties of straw, cereals, canola, sunflower and maize carried out by a research institution. The aggregate ERR calculated under these assumptions was 88 %. There are questions about the validity of the analysis:
(i) Although the project included technologies for several crops, the calculations included canola only, it was not explained why only canola was selected.
(ii) It was not clear how the gross margins (incremental benefits) were calculated, and what the assumptions were for calculating the project and non-project yields, operational costs and revenues.
(iii) Furthermore, the ERR calculation compared project beneficiary village's size of canola fields with the non-project village's canola fields to come up with the incremental canola areas, and the incremental benefit per ha was assumed constant. The assumption was that the canola areas increased with the help of the project, but with a decreasing trend each year. However this method ignored the fact that as the canola areas in with-project village increased, the without-project village did not probably leave its fields idle, but cultivated other crops and therefore had benefits coming from those other crops. Ignoring this additional benefit coming from other crops, probably led to overestimating the total incremental benefits.
(iv) The example project yielded an ERR of 98 % but the ICR did not provide ERR results for the remaining 31 projects.
Administrative and Institutional Efficiency
The project had a number of administrative and operational inefficiencies. Frequent changes in government and Ministers of Agriculture and the insufficient focus on priorities in Ministry of Agriculture and Rural Development negatively affected overall project implementation. The continuous non-performance of The National Veterinary and Food safety Authority and the lack of any concrete plan to remedy this situation led to the decision to discontinue all the activities, and led to the cancellation of important activities in food safety. Furthermore, the project suffered occasionally from insufficient allocation of government funds and in many instances from delayed allocation of funds for the project activities, which substantially affected implementation throughout the project’s life. Unfilled vacancies within the PMU affected some of the PMU responsibilities and project implementation (i.e. hiring of M&E expert). As a result of these problems, the project closing date was extended for approximately 2.5 years from September 30 2010 to March 25, 2013. In addition, these project delays and inefficiencies led to the cancellation of 34% of the project loan.
Efficiency of the project is rated modest
a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:
* Refers to percent of total project cost for which ERR/FRR was calculated