Measuring Results of the Armenia Irrigation Infrastructure Activity

The MCC Compact with Armenia was a five-year investment (2006-2011) of $176.6 million. The

$121.5 million Irrigation Infrastructure Activity is the subject of an independent impact evaluation summarized here. The evaluation also includes analysis for two Sub-Activities under the Water-to-Market Activity, which follows-up on the Water-to-Market Activity evaluation previously completed in March 2013.

  • The Irrigation Infrastructure Activity rehabilitated irrigation infrastructure such as pumping stations, canals, conversion to gravity systems, drainage, and tertiary canals in order to increase the land area under irrigated production and to improve the overall efficiency of sourcing and delivering water to farmers. The Water-to-Market Activity complemented the Irrigation Infrastructure Activity by providing farmer training, credit, post-harvest/processing/marketing assistance, and institutional strengthening of irrigation management. Together these activities were designed to transition farmers to growing high-value agriculture (HVA) and increase income.
  • While farmers near tertiary canals perceived improvements in the timeliness and reliability of irrigation water, others near rehabilitated large infrastructure did not. There was also no evidence that farmers irrigated more land, that production of HVA crops or their yields increased, or that household income and consumption increased.
  • For communities that received Water-to-Market training in 2008, the evaluation did not find that adoption of on-farm water management practices was substantially higher for farmers in 2013 than it was in 2010. The levels of adoption of profitable new practices remained low. In addition, Water User Associations’ financial status have leveled off since 2010, with little additional progress toward financial self-sustainability.
  • Two key lessons learned from this evaluation include the following: (1) root cause analysis is critical to successful project design, especially where behavior change is integral to the program logic and (2) sector-specific technical capabilities should be required on the evaluation team when needed to assess key intermediate outcomes.
  • This evaluation is complete and there are no planned next steps.

In Context

The MCC Compact with Armenia was a five-year investment (2006-2011) of $176.6 million in two projects: the Irrigated Agriculture Project and the Rural Road Rehabilitation Project. The Irrigated Agriculture Project included two major activities, Irrigation Infrastructure and Water-to-Market (WTM). The $121.5 million Irrigation Infrastructure Activity is the subject of an independent impact evaluation, the results of which are summarized here. This component represents 69 percent of the total Compact funds. The independent evaluation also follows up on previous analyses conducted on two components of the WTM Activity, which as an activity represents 18 percent of the Compact funds. Other components of the Compact are the subject of previously released independent evaluations.

                 

 

Program Logic

The Irrigation Infrastructure Activity was designed to rehabilitate irrigation infrastructure, such as pumping stations, canals, conversion to gravity systems, drainage, and tertiary canals in order to increase the land area under irrigated production and to improve the overall efficiency of sourcing and delivering water to farmers. The anticipated short-term outcomes of the Irrigation Infrastructure Activity were to improve the efficiency of irrigation and to increase the area of irrigated land—both by expanding irrigation to areas that were not irrigated before the project, and to prevent decreases in irrigated area in the absence of the project as rehabilitated irrigation equipment deteriorated further. With access to a more consistent supply of irrigation water, farmers could increase their agricultural production, shift crop production toward higher-value agriculture (HVA), increase sales, and increase agricultural profits.

 

                  

 

There were several key assumptions underlying the Irrigation Infrastructure program logic during the design of the investment:

  • Farmers already had reliable irrigation water through existing structures or would gain access through the Irrigation Infrastructure Activity
  • Content and duration of training and technical assistance provided through the Farmer Training Sub- Activity would be sufficient to improve knowledge of targeted farmers and enterprises, triggering behavior change due to high education levels of farmers
  • Appropriate selection would be conducted of farmers and enterprises to participate in the Farmer Training Sub-Activity
  • Credit offered through the Credit Sub-Activity would be provided for a small percentage of trained farmers and in addition, would catalyze increased agriculture lending through existing finance mechanisms and enable farm investments
  • Sufficient access to markets would be facilitated through existing mechanisms or the Post-harvest Processing and Marketing Sub-Activity
  • Adoption of new, improved agricultural practices would lead to an increase in farm productivity
  • Increase in farm productivity, along with improved marketing channels, would lead to an increase in farm income
  • Increase in farm income would lead to an increase in overall household income

For a more detailed version of the program logic, please refer to page 5 of the final evaluation report.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes. When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated activity.

Indicators

Level

Baseline (2006)

Actual Achieved (Sept 2011)

Target

Percent Complete

Government budgetary allocations for maintenance of irrigation system (Armenian drams)

 

Outcome

 

N/A

 

1,385,538

 

1,500,000

 

92.4%

Primary canals rehabilitated (kilometers)

Output

0

41.8

41.8

100%

Primary canal structures constructed

Output

0

267

244

109.4%

Pumping stations renovated

Output

0

17

17

100%

Gravity canals rehabilitated (kilometers)

Output

0

49.5

44.5

111.2%

Conversion from pumping to gravity completed

Output

0

5

4

125%

Tertiary canals rehabilitated (kilometers)

Output

0

232.8

220

105.8%

Drainage systems rehabilitated

Output

0

13

13

100%

Drainage canals cleaned (kilometers)

Output

0

470

470

100%

Source: Closeout ITT from Sept 2011, which includes data through the end of the Compact, based on reporting from MCA-Armenia, construction companies, and the State Water Committee.

 

The average completion rate of output targets is 106 percent and targets were met or exceeded in 8 of the 8 output indicators. The average completion rate of outcome targets is 92 percent and the target was met or exceeded for the only outcome indicator. 1

Evaluation Questions

The evaluation was designed to answer the following questions:

  • Did the program affect the quantity and reliability of irrigation water provided to Armenian farmers? The analysis controls for the provision of WTM training, so these estimates are focused on the influence of the Infrastructure Activity.
  • Did farmers adopt new agricultural practices as a result of the program? Further, is there evidence that farmers who received WTM training postponed adopting new agricultural practices until after the irrigation infrastructure had been rehabilitated?
  • Did the program affect agricultural productivity?
  • Did the program improve household well-being for farmers served by the rehabilitated infrastructure, especially income and poverty?
  • Is there evidence that the infrastructure investments will be sustained after rehabilitation was complete? Are the Water User Associations (WUAs) themselves financially sustainable? This is the key outcome considered in reassessing the longer-term effects of the Institutional Strengthening Sub-Activity.

Evaluation Results

The Irrigation Infrastructure Activity evaluation included two components – one for tertiary canals and one for large infrastructure (pumping stations, gravity schemes, and main canals). The tertiary canal evaluation and survey were designed first and then at MCC’s request, Mathematica Policy Research designed an evaluation for the large infrastructure using the existing tertiary canal survey data that had already been collected in the communities expected to benefit from the large infrastructure rehabilitation. The evaluation also includes follow-on analysis that was not originally planned for the WTM Farmer Training and Institutional Strengthening Sub-Activities, both of which were previously evaluated in March 2013. 2

Irrigation Infrastructure

Evaluator

Mathematica Policy Research

Impact or Performance?

Impact

Methodology

Matching

Evaluation Period

Implementation start and end:  December 2008 to September 2011 Baseline data collection: Early 2010 covering the 2009 agricultural season

Final data collection: Early 2014 covering the 2013 agricultural season

Evaluation exposure period: 2 to 3 years after irrigation rehabilitation completion

Outcomes

  • Tertiary  Canals  only:  Farmers  perceived  improvements  in  the timeliness and reliability of irrigation water.
  • Large Infrastructure only: Farmers were more likely to report dissatisfaction with the timeliness and quantity of irrigation water.
  • Farmers did not irrigate more of their land or more frequently overall, but farmers increased the total hours of irrigation on some portions of their land.

Objective-level Outcomes

  • No evidence of increases in production of HVA crops or yields (productivity)
  • No evidence of increases in sales or agricultural profits

Effect on household income attributable to MCC

  • No evidence of increases in household income or consumption
  • No evidence of decrease in the rural poverty rate

 

 

Farmer Training Follow-up

Evaluator

Mathematica Policy Research

Impact or Performance?

Performance

Methodology

Ex-post

Descriptive comparison between farmers in the WTM survey in 2010/2011 and farmers in the Tertiary Canal Survey (TCS-WTM) in 2013/2014

Evaluation Period

Implementation start and end:  Training occurred from 2007 to 2011; however the sample included in the evaluation received training in 2008

Baseline data collection: Early 2008 covering the 2007 agricultural season through the WTM survey

Interim data collection: Early 2011 covering the 2010 agricultural season through the WTM survey

Final data collection: Early 2014 covering the 2013 agricultural season through the TCS-WTM survey

Evaluation exposure period: Approximately 5 years after training

Outcomes

  • Adoption rates of on-farm water management practices did not systematically increase over time
  • Evidence of higher adoption rates for improved soil preparation and improved post-planting procedures
  • Rates of cultivation between the two samples of farmers were comparable for most crops

Objective-level Outcomes

  • N/A

Effect on household income attributable to MCC

  • N/A

 

Institutional Strengthening of Irrigation Management Entities Follow-up

Evaluator

Mathematica Policy Research

Impact or Performance?

Performance

Methodology

Pre-Post

Evaluation Period

Implementation start and end:  September 2008 to October 2011 Baseline data collection: 2007 agricultural season

Interim data collection: Annual from 2008 to 2012

Final data collection: Administrative data from 2013 and WUA Director interviews in early 2014 covering the 2013 agricultural season

Evaluation exposure period: 2 years after the intervention ended

Outcomes

  • No large differences between 2007 and 2013 in WUA water deliveries, water losses, or total water intake from water supply agencies
  • Increase  in  the  water  payment  rate  achieved  after  activity completion was then sustained in 2011, 2012, and 2013
  • Average WUA deficits decreased by 35 percent since 2007
  • WUAs have not met the targeted cost recovery rates in any year after the activity ended, and the improvements in the cost recovery rates have leveled off since 2010

Objective-level Outcomes

  • N/A

Effect on household income attributable to MCC

  • N/A

Lessons Learned

  • Root cause analysis is critical to successful project design, especially where behavior change is integral to the program logic. The Irrigated Agriculture Project did not result in the increases in high- value crops that were projected during project design. This could either be because the root causes of farmers’ failure to increase production of high-value crops were misdiagnosed (i.e. the problem was not water, training or credit) or the project as implemented did not adequately address those causes (i.e. improvements to the irrigation system were not sufficient to actually increase access to water and increases in irrigated land or training of only a subset of farmers receiving improved irrigation was insufficient). It’s important that the program logic developed during project design be based on evidence about what the root causes of a problem are and then implementation should align with that program logic.

 

  • Supporting the development of new institutions, such as Water User Associations, is inherently difficult and can be a long-term undertaking, so MCC projects should account for the five year timeline and anticipate necessary follow-on activities. While MCC’s intervention improved WUA performance during the Compact, WUAs did not continue to improve their cost-recovery after the Compact ended. The Compact did not put in place the necessary environment for WUAs to continue to improve and become self-sufficient. Future programs should focus on building the right environment for sustainability from the beginning by clearly defining reasonable achievements within a five-year timeframe and planning for what will need to occur after.

 

  • During a project re-scoping, the program logic, economic analysis, potential beneficiaries, and evaluation plan should be re-assessed in a cohesive way by a coordinated project team. The Irrigation Infrastructure Activity was re-scoped; however, the other complementary activities were not assessed in the same way at the same time. This contributed to a disjointed project during implementation and may be one of the reasons for the lack of impact. In addition, the re-scoping economic analysis, on which the design of the evaluation was based, may have been overly optimistic about the behavioral changes that would occur from the project. An integrated, cross-sectoral review of the economic analysis may have resulted in more realistic assumptions of behavioral changes.

 

 

  • Better water monitoring tools could help Water User Associations while also providing better measurement of outcomes. Improved water delivery and reduced losses were fundamental expected outcomes of the Irrigation Infrastructure Activity, but the existing data have not been validated, and there are not accurate measures of water delivery to farmers. This makes it difficult to know whether water availability has actually improved for individual farmers. Creating the means for WUAs to more precisely measure water delivery would also potentially help them manage their resources more effectively.

 

  • An irrigation project where demand for the intervention exceeds available funding, can provide an opportunity for random assignment (or other allocation mechanisms that are fair, efficient and informative) that can improve the level of rigor of evaluation results. In cases like this where demand for irrigation outstrips funding, a well-executed randomized controlled trial of improvements like the tertiary canals could be structured so that impacts can be rigorously evaluated for subgroups of farmers that receive different types of infrastructure rehabilitation.

 

  • Sector-specific technical capabilities should be required on the evaluation team when needed to assess key intermediate outcomes. The evaluation team did not include irrigation infrastructure expertise and therefore, could not assess the actual state of the irrigation system in 2013. The post- project state of the infrastructure is an important aspect of assessing project results. Infrastructure evaluations should be designed to include a technical assessment of the improved infrastructure to ensure that it is functioning as envisioned after the compact.

 

  • Evaluation questions are based on the program logic and must be designed carefully from the beginning to understand the scope and limitations of the evaluation. Given that the Irrigated Agriculture Project was not designed and implemented as a package of coordinated interventions for a targeted group of beneficiaries, MCC could not design an evaluation of the overall Project. Even though the individual Activity evaluations were informative, the lack of coordination between Activities limited MCC’s ability to report on the overall impact of the Project. In the future, MCC should work with all stakeholders to understand the program logic, how the program will be implemented and clarify what the evaluation will be able to answer and not answer from the beginning. In addition, MCC should design evaluations to be able to explain “why” if the expected impacts do not materialize.

 

MCC has learned from these lessons and is working on incorporating them into how compacts are designed, implemented, and evaluated in these ways:

  • MCC’s revised guidance for compact development requires a problem diagnosis phase immediately after the constraints analysis. During this phase, the key root causes of the binding constraints are analyzed in order to ensure sufficient understanding of the problems that the project will be designed to address. The next phase of the development process includes building strong project logics for the proposed compact program.
  • MCC’s policy on compact modifications requires that the team economist review any potential change in scope to assess its impact on the expected economic benefits and beneficiaries.

 

  • Evaluation scopes of work are including more technical sector expertise on the evaluation teams. In addition, many evaluations are being designed to answer “why” when expected results to do not materialize.

 

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes