Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Irrigation Distribution System & Management Improvement Project


  
1. Project Data:   
ICR Review Date Posted:
04/12/2012   
PROJ ID:
P008286
Appraisal
Actual
Project Name:
Irrigation Distribution System & Management Improvement Project
Project Costs(US $M)
 39.1  44.1
Country:
Azerbaijan
Loan/Credit (US $M)
 35.0  39.7
Sector, Major Sect.:
: Irrigation and drainage, General public administration sector
Cofinancing (US $M)
   
Theme(s):
Rural services and infrastructure (33% - P) Rural policies and institutions (33% - P) Water resource management (17% - S) Decentralization (17% - S)      
L/C Numbers:
C3789
Board Approval (FY)
  2003
Partners involved
No 
Closing Date
03/31/2010 09/30/2010
         
Evaluator: Panel Reviewer: ICR Review Coordinator: Division:  
Keith Robert A. Oblitas
Ridley Nelson IEG ICR Review 1 IEGPS1

2. Project Objectives and Components:

a. Objectives:
To improve the effectiveness and financial viability of irrigation water distribution and management on 56,000 ha, through development of Water User Associations and the Amelioration and Irrigation Joint Stock Company. (Source PAD; the Credit Agreement is substantially similar.)

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components (or Key Conditions in the case if Adjustment Loans):
The Irrigation Distribution System and Management Improvement Project (IDSMIP) had 3 components:

1. Development of Water User Associations (WUAs) on 56,000 ha ($7 million appraisal estimate; $3.8 million actual) through 3 activities:
(a) Development of the capacity of the Amelioration and Irrigation Joint Stock Company (AIOJSC) including establishment, with core specialist staff, of a Central Support Unit within AIOJSC as its operational unit and a Project Implementation Unit specifically for coordinating the project; and establishing 11 "raion" (district) support units, each with an engineer, water management specialist and WUA support specialist.
(b) Training of WUAs through information dissemination and public awareness of farmers and local and central governments; development of training modules and a subsequent training program for WUAs covering technical, social, procedural and financial aspects of WUA management, and preparation of WUA irrigation services plans with the objective of transferring management and authority at the on-farm irrigation system level from AIOJSC to the WUAs.
(c) revision of the Law on Amelioration and Irrigation to change the status of WUAs from for-profit limited liability companies to non-profit associations with limited tax liability and voluntary membership.

2. Irrigation and Drainage Infrastructure Rehabilitation ($30 million appraisal estimate and $37,4 million actual. Rehabilitation of on-farm infrastructure on 56,000 ha, installation of 1,000 hydroposts, selective rehabilitation at secondary levels and emergency rehabilitation of head-works serving 140,000 ha.

3. Project Management support ($2.1 million appraisal estimate; $2.9 million actual) through continued financing of a Project Implementation Unit (PIU) that had been established under the Bank financed irrigation project (the Rehabilitation and Completion of Irrigation and Drainage Infrastructure Project) that had preceded IDSMIP.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
There were no changes in project objectives and monitorable indicators, and the Credit of SDR 25,7 million was fully disbursed. There was a restructuring in March 2010, just before the originally proposed project closing date. The restructuring: first, extended project closure by an extra six months, in order to provide the Project Implementation Unit (PIU) with more time to evaluate the project and lessons learned and incorporate these in the design of IDSMIP's follow-on project. Second, it was decided not to finance heavy equipment for WUAs and the disbursement schedule was adjusted to apply the resultant savings on the goods category to civil works and consultants. The reason for the reallocation was because of higher than expected cost increases in both of these categories, especially for civil works.
There were some differences between intended and actual irrigated areas and the intended and actual number of WUAs. The project covered 52,000 ha or 93 percent of the initially intended 56,000 ha. The slightly reduced area was primarily because of shortage of funds due to higher than expected inflation of the costs of materials (refer section 2 d). Also, to follow hydraulic boundaries more closely, 22 larger WUAs were actually established compared with the 45 WUAs projected at appraisal.)


3. Relevance of Objectives & Design:

Relevance of Objective: High
The project objective was highly relevant and remains so. When the project was being prepared in 2002-2003, agricultural GDP in Azerbaijan was only just emerging from the free-fall in value added (about a 40 percent decline) following the Soviet period. Agricultural productivity needed to grow well beyond Soviet period productivity for several strategic reasons: (i) agricultural employment accounts for 40 percent of the work-force, and most of Azerbaijan's poor live in rural areas; (ii) the sector is an economic mainstay. It contributes 30 percent of the country's non-oil GDP and about half of non-oil exports; and (iii) with oil reserves and production anticipated to decline over the next 10 to 15 years, agriculture will have to be a key contributor to taking up the slack in the post-oil boom to enable robust growth of Azerbaijan's economy. As rainfall on most agricultural land in Azerbaijan is less than 500 mm, irrigation is an important enabler of higher yields and crop diversification. Yet irrigation infrastructure had received little Government funding and had deteriorated markedly since Independence. Moreover, farmers lacked the organizational and technical skills to manage irrigation systems themselves. Thus, IDSMIP's combination of infrastructural improvement and assistance developing WUAs and strengthening the apex organization for implementing the WUA program (the Joint Stock Company) responded to a key need. The issues described above are well recognized in the two most recent strategy documents for Azerbaijan - the 2009 Country Economic Memorandum and, most recently, the 2010 Country Partnership Strategy for FYs 11-14. Likewise, the 2000 and 2003 Country Assistance Strategies and the 2003 Poverty Reduction Strategy Paper (the strategy documents prepared at the time of IDSMIP's preparation and appraisal) had agriculture as one of the lending priorities - three (including IDSMIP) out of the 12 projects proposed in the PRSP were agricultural.

Relevance of Design: Substantial.
  • IDSMIPs design was a good match to the project's objectives. Water distribution would be improved by the combination of system rehabilitation and better management under the WUAs. Agricultural productivity could be expected to increase providing better incomes with which to pay WUA fees for operations and maintenance. AIOJSC and the district support units could provide hands-on advice to the WUAs.
  • Links with agriculture. The design would have been better if there had been a systematic linkage of IDSMIP with the agricultural extension activities of the Agricultural Development and Credit Project. If ADCP and IDSMIP had harmonized their activities in locations with WUAs, so that ADCP's agricultural extension and training activities made sure that all WUAs were in the extension program, productivity impact would likely have been better. The lack of arrangements between ADCP and IDSMIP continued throughout IDSMIP's implementation, and the follow-on project's PAD also has no references to such coordination.

4. Achievement of Objectives (Efficacy) :

IDSMIP's efficacy can be considered under the two principle objectives of the project, which refer respectively to: (i) the effectiveness and (ii) the financial viability of irrigation. A related output objective "provision of support to Water User Associations and the State Amelioration and Irrigation Committee" (AIOJSC) also referred to in the project's objectives, were means to these outcomes. The ICR reports that, based on the final and baseline surveys, output results as compared with targets for the Monitorable Indicators were:
(i) Efficacy of improving the effectiveness of irrigation (Rating Substantial):
The WUAs and irrigation system performed as targeted under the Monitorable Indicators:
Monitorable Indicators I. Some 91 percent of O&M was carried out according to the O&M plans. The PAD targeted 90 percent, and the baseline situation was no maintenance. (percentages are expressed as combined actual O&M costs compared with combined planned O&M for all 22 WUAs combined).
Monitorable Indicators II. 78 percent of farmers in the 22 WUAs were satisfied with WUA managerial and operational performance (The PAD aimed for 75 percent).
Monitorable Indicators III. Number of irrigations. An average of 82 percent of farmers received four irrigations for summer crops, the same as the PAD target (The base situation was 2 or less irrigations); and:
Monitorable Indicators IV. 208 out of the 213 re-registered WUAs (combined total of WUAs with rehabilitated systems and WUAs without rehabilitation) have taken over the management of their systems (targets were for all WUAs to have taken over the systems).

The Monitorable Indicators only partly convey the results in terms of effectiveness (the objective), in particular as concerns productivity. However, other data reported in the ICR are positive, and, based on survey data, the combined impact on productivity of WUA formation and system rehabilitation has been good: (i) a major factor was the near doubling of the area of land returned to production compared with the appraisal assumptions; (ii) cropping patterns have also changed, with increases in the proportion of crop area under horticulture and reduced area under cereals; and (iii) yields for farmers in rehabilitated WUAs have increased by between 15 to 30 percent from 2006 to 2009 compared with yield increases in the same period of less than 10 percent on non-rehabilitated irrigation systems. In turn, these substantial gains have provided a strong base for the project's efficiency (Section 5).

(ii) Efficacy of improving the financial viability of irrigation (Rating Modest).
The ICR comments that irrigation service fees increased about 3 to 5 times compared with pre-rehabilitation rates. Also, the central irrigation and WUA coordination and support unit (the Amelioration and Irrigation Joint-Stock Company, AIOJSC) became financially independent from government, based entirely on contributions of the WUAs (prior to IDSMIP only 10 to 20% of its costs were financed by WUAs). This is an important achievement. AIOJSC provides the central services needed by the WUAs and its financial independence from government provides better assurance that its finance needs will continue to be provided.
As concerns WUAs, at present, those with rehabilitated systems carry out an average of 90 percent of O&M as determined in the O&M plans. Revenues are thus close to required O&M expenditures. However, the WUAs' financial viability in the longer term (beyond about years 5 to 10) is not yet assured. Maintenance needs will grow over time as the systems get older. The ICR (Annex 3) estimates that average service fees for WUAs are now about 40 percent of engineering estimates of average longer-term needs for O&M. Thus, while the financial viability of irrigation has improved, revenues are still below appraisal targets and have progressed to only 40% of the ultimate need for revenues to fully cover the larger O&M requirements in the longer term. Forty percent is a significant improvement on the near zero collections before IDSMIP but is still only half of appraisal targets and lower still than ultimate needs.
The task team has advised that the follow-on project to IDSMIP (The Water Users Association Development Support Project, approved in April 2011), will continue to support the WUAs formed under IDSMIP, and that this attention is expected to raise the financing percentage. Nevertheless, while due account can be taken of the fully satisfactory financial achievement concerning WUA contributions to AIOJSC; the transformation of AIOJSC to a financially independent entity; the negligible cost recovery of WUAs when the project began; and the further improvements in WUA collections expected under the follow-on project, IDSMIP has made only partial headway towards the long-term financial viabllity of WUAs. IDSMIP's efficacy for the financial viability objective is thus assessed Modest.

5. Efficiency (not applicable to DPLs):

Rating Substantial .
Benefits for farmers have been considerable. The ICR reports that average farm incomes more than doubled for farmers in the rehabilitated irrigation schemes.
As concerns the ERR, the recalculated rate at project completion of 33% was appreciably higher than the appraisal estimated ERR of 21%. The main reasons were the increase in area returned to production compared with the area assumed at appraisal, increased yields, and crop diversification to higher value crops.
The ICR's otherwise thorough economic analysis does not estimate the sensitivity of the ERR to different benefit streams. This gap is regrettable as it does not allow testing of the ERR under different levels of maintenance and assumed productivity - inadequate future maintenance and hence declining infrastructure is the project's main risk. (Section 7). However, given the much higher incomes under irrigation, WUAs would have strong financial incentives to not allow systems to deteriorate to the extent that their incomes are significantly reduced.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:



Rate Available?
Point Value
Coverage/Scope*
Appraisal:
Yes
21%
100%
ICR estimate:
Yes
33%
100%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Outcome was Moderately Satisfactory.
IDSMIP's objectives were Highly relevant to Azerbaijan's need to expand agricultural productivity. Agriculture was the country's main driver for employment and poverty alleviation and was the primary contributor to non-oil GDP and exports. Improving the irrigation sector's productivity and financial viability was, thus, key to enhancing economic growth and rural welfare. An improved irrigation service could provide a base for significantly higher agricultural growth. The project's design was Substantially relevant. It was simple, focusing on two thrusts covering the two action areas best suited to support the objectives, and was accompanied by an outcome oriented results framework.
The project's efficacy was Substantial for the objective to improve the effectiveness of irrigation. All output targets were met and agricultural productivity increased appreciably. But for the second objective - improving the financial viability of irrigation - achievement in covering O&M costs was incomplete - service fees increased several fold, but at project completion were still only 40% of estimated long-term annual O&M costs. The higher financial contributions of WUAs to the central WUA agency exceeded expectations and made it financially independent from Government - a good achievement. But for WUAs, achievements were partial. The efficacy of this objective was Modest.
IDSMIP's efficiency was Substantial. The doubling of farm incomes was a significant achievement and it also contributed to a re-estimated ERR at completion of 33%. While there is a risk that these very satisfactory achievements could be eroded over time if WUAs provide insufficient funds for O&M (Section 7), a countervailing factor is the strong financial incentives for farmers to fund and maintain their systems in order to continue obtaining the higher productivity and incomes that the project had enabled.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating :

Significant. The main risk is at WUA level and concerns the degree to which WUAs will enhance water fees sufficient to cover O&M (Refer Sections 4 and 6). Until this is achieved, the sustainability of the rehabilitated systems is uncertain. However, a countervailing force is likely to come from the self-interest of farmers. They can be expected to want to maintain and even improve the two-fold increase in incomes that IDSMIP has enabled. Thus, the motive to protect their incomes by up-keeping the irrigation systems would appear to be strong.
The relative costs of maintenance compared with farm incomes suggests that affordability is not a constraint for full financing of O&M. Engineering estimates reported in the ICR are that O&M costs would be about $18 per hectare. The increment in net margins as a result of irrigation are estimated in the economic annex to be about $800 per hectare. Thus, water fees sufficient for full O&M, would be only 2 percent of the increase in net farm incomes that IDSMIP has enabled, Nevertheless, for the moment, water fees are still insufficient for full maintenance, Given this, the risk to development outcome is rated Significant.

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

Quality at Entry: Rating Satisfactory
The task team recognized the key priorities for the irrigation sub-sector and designed a project well suited to achieve these objectives. The Bank took account of Azerbaijan's limited capacity in irrigation management at that time, by designing a simple project with a sharp focus on community management of irrigation. The simplicity and focus of the project on only two main activities - WUAs and rehabilitation - was appropriate for a country with little experience in WUAs and participatory irrigation management. Within this focused structure, however, a comprehensive approach to WUA management was taken which paid dividends in the subsequent largely successful development of these institutions.
There was an outcome oriented results framework and, in most respects, a good set of outcomes oriented monitorable indicators, which were subsequently used to monitor project progress throughout implementation and at completion.
Supervision: Rating Satisfactory.
Supervision missions were regular (about twice yearly) and pro-actively oriented to resolution of implementation issues. The task team included the specializations needed and the ICR reports good back-up from the country office. Dialogue with government on the politically sensitive WUA cost recovery issue was intensive, albeit with only partial results.
Towards the end of the project the task team identified and then prepared a follow-on project which was approved in early 2011. This was appropriate given the long-term nature of WUA development and operations. Retrofitting a gender study at the end of the project was a a good initiative. While too late to benefit IDSMIP, the study has potential to be a useful base for gender inclusion in the follow-on project.
Both at entry and during supervision of IDSMIP, the Bank could have considered making arrangements for the Agricultural Development and Credit Project's extension services to include IDSMIP's WUAs and their specific needs (Section 3). While this was a missed opportunity (for both the IDSMIP and ADCP teams), in most respects the Bank's performance was good, both in quality at entry and during project implementation. The Bank's overall performance is rated Satisfactory.

a. Ensuring Quality-at-Entry: Satisfactory

b. Quality of Supervision: Satisfactory

c. Overall Bank Performance: Satisfactory

9. Assessment of Borrower Performance:

Government Performance: Rating Satisfactory.
The ICR reports strong political support and provision of adequate counterpart funds throughout project implementation. This is particularly significant given that, based on IEG observations on the Azerbaijan rural sector, provision of adequate counterpart funding for Bank agricultural projects has more usually been constrained. AIOJSC and its Central Support Unit were effective in bridging between Government and the Project Implementation Unit, and access to senior government officials was usually good. AIOJSC became financially independent due to increased contributions by farmers.
Government might, however, have considered from its overview vantage for all agricultural development, the opportunities available for better linking of irrigation and agronomic improvements (Section 3).

Implementing Agency Performance: Rating Satisfactory.
Although earlier irrigation projects had provided some experience with WUAs, IDSMIP was particularly intensive in its comprehensive training and WUA advisory services. Achieving IDSMIP's comprehensive approach required significant further learning and adaptation by the Project Implementation Unit and field staff as experience was gained. IDSMIP met its physical output targets, which were demanding in terms of organization, experimentation and field supervision. While WUAs made limited progress increasing their revenues for O&M, the primary constraints were cultural or local politics in nature rather than due to lack of motivation and awareness building by the implementing agency.

a. Government Performance: Satisfactory

b. Implementing Agency Performance: Satisfactory

c. Overall Borrower Performance: Satisfactory

10. M&E Design, Implementation, & Utilization:

Design
A good foundation was provided in the results framework and its associated monitorable indicators. These were outcome oriented and provided useful proxies with which to assess project impacts. A complication was that the number of indicators (27) went beyond the five outcome indicators specified in the PAD. While these data appear to have provided useful measures of project progress and achievements, separating the outcome indicators from other more management information measures would normally engender greater attention to project objectives.
Implementation
The baseline survey was undertaken late - in 2006, three years after Board approval. However, the task team has advised that the survey was undertaken by mid-term review and before any infrastructure had been completed. Thus, while delaying a baseline survey well into a project's implementation is not good practice, in this particular case results would still have broadly measured parameters at project start-up. The ICR contains few details on how M&E was implemented although there is reference to surveys for some data. Nevertheless, the availability of data for the ICR to assess achievement of the monitorable indicators, and the extensive information on productivity in the ICR's economic analysis annex indicates that substantial relevant information was obtained during project implementation.
Utilization
The ICR reports that data was presented to management regularly, was used in meetings, and resulted in a number of operational changes in project implementation.

a. M&E Quality Rating: Substantial

11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts):

On environmental and social issues IDSMIP was a category B project. Various studies were done during preparation and implementation of IDSMIP and the ICR assesses that no major issues were encountered. Better water distribution and drainage would typically be environmentally beneficial in reducing waterlogging and salinization. Social impacts were considered positive as water would be more available for the generally poorer tailenders.
Impacts on women were not reviewed until the end of the project when the task team hired a consultant to carry out a gender study. This pro-active initiative was, however, too late for the IDSMIP project period.
(The task team has advised that a more structured social inclusion program will be built into IDSMIP's follow-on project. Although details of a gender program are not included in the PAD, follow-up consultancy work is planned which may provide a base for retrofitting gender features into the project. Integration at the time of project preparation would have been better.)
The ICR reports that there were no fiduciary issues.


12. Ratings:

ICR
IEG Review
Reason for Disagreement/Comments
Outcome:
Satisfactory
Moderately Satisfactory
One of the project's two objectives - financial viability of WUAs - was partially met. 
Risk to Development Outcome:
Moderate
Significant
The rehabilitated irrigation systems are at risk until the fees collected by WUAs cover their O&M costs. 
Bank Performance:
Satisfactory
Satisfactory
 
Borrower Performance:
Satisfactory
Satisfactory
 
Quality of ICR:
 
Satisfactory
 

13. Lessons:

IDSMIP exemplifies good practice for irrigation improvement in most of its design and implementation features, but there are several areas where there is room for improvement. Amongst the lessons from the project experience the following are particularly relevant for Azerbaijan, with possible applicability in other countries as well (Lessons 1, 2 and 3 are from the ICR, and lessons 4 and 5 have been added based on issues discussed in this ICR Review):

From ICR:
  1. Irrigation improvement for management by WUAs needs, as a central objective, the collection of revenues by WUAs such that they can fully cover their O&M needs. Key to sustaining the rehabilitated works and the gains in agricultural productivity.
  2. WUA development is a long-term process. IDSMIP's experience with WUAs shows that further capacity building and technical advice is still needed for the WUAs, and above all, WUA revenues and O&M need to be brought to adequate levels, key challenges for IDSMIP's follow-on project.
  3. Several good-practice features were illustrated by IDSMIP. Amongst these were:
- Simplicity and focus helped IDSMIP to concentrate on development of WUAs. The project was limited to institutional development (WUAs and support institutions), and selective rehabilitation of the WUAs' irrigation and drainage systems. The focus enabled concentrated attention on these two thrusts.
- Community based irrigation rehabilitation should emphasize institutional development first. IDSMIP's priority was on development and capacity building for WUAs. Rehabilitation of a system was implemented only after the WUA had developed at least a starting-level capacity for subsequent management.
- Emphasize training and capacity building. Training and advisory activities for the WUAs were given particular priority. Study tours in the first year of the project helped build knowledge and enthusiasm amongst the support institutions and WUAs.

Added by IEG:
4. Link irrigation improvements with enhanced agricultural extension services. There would likely have been greater impact on agricultural productivity if the agricultural and irrigation projects had been mutually supportive.
5. Gender and social inclusion needs should be assessed during project preparation and relevant actions built into project design. Gender issues were considered at the end of the project when a study was commissioned. If the study had been done earlier, it might have influenced some adjustments in WUA operation plans and in the training program.

14. Assessment Recommended?

Yes
Why?
As part of a review of several project/country experiences to provide lessons for future community based irrigation improvement programs.

15. Comments on Quality of ICR:

The ICR is thorough, lessons-oriented and forward looking, and the text is in general well organized to place material in the relevant sections of the ICR template. The economic analysis Annex is particularly informative on productivity related data. The discussion of adjustments needed in the WUA program subsequent to IDSMIP is thoughtful and candid, although how the follow-on project will handle these adjustment needs could have been made clearer.
There were several gaps: (i) while the economic analysis is thorough, there was no sensitivity analysis of the effect on farm incomes and the ERR of a decline in benefits due to deteriorating infrastructure. This is the project's primary risk; (ii) data sources are not always provided; and (iii) discussion in the text of the project's efficacy (Section 3.2) would have been better focused if it had been more structured on the project's objectives. A more evidence-based discussion, referring to some of the facts in the earlier discussion of achievements by components, and in Annex 2, would have been possible. For instance, in Section 3,2 (efficacy), and in the choice of Monitorable Indicators, there could have been more discussion of institutional outcomes. In fact, the generally good choice of MIs, missed an assessment of institution strengthening achievements, despite the implied emphasis placed on institutional development in the project objectives. Especially when points affecting the achievement of a project's objectives are key to the assessment of project outcome, the evaluation need not be restricted only to the Monitorable Indicators identified during project preparation. Nevertheless, given the strengths of other aspects of the ICR as noted above, the ICR is rated Satisfactory overall.

a. Quality of ICR Rating: Satisfactory

(ES-Rev5-Jul/06)
© 2012 The World Bank Group, All Rights Reserved. Terms and Conditions