Summary

July 24, 2014

Measuring Results of the Armenia Farmer Training Investments

In Context

The MCC compact with Armenia was a five-year investment (2006-2011) of $177.6 million in two projects: irrigated agriculture and rural road rehabilitation. The Irrigated Agriculture Project included two major activities, irrigation infrastructure and water-to-market (WTM). The WTM activity consisted of four components: on-farm water management and high-value agriculture farmer training; post-harvest, processing and marketing enterprise training; access to credit; and institutional strengthening of water users associations. The $14 million on-farm water management (OFWM) and high-value agriculture (HVA) farmer training component is the subject of both the results described here and an independent impact evaluation released by MCC in October 2012. This component represents 8 percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.

Program Logic

The Irrigated Agriculture Project was designed to address the physical, managerial and financial investments needed to generate sustainable increases in rural incomes through irrigated agriculture. The WTM activity was designed to complement the infrastructure investment by training farmers on new technologies and shifting production to higher-value crop production. The program logic assumed that trained farmers would benefit from improved water resources, improved marketing opportunities and access to credit for investment. Combined with training, these aimed to provide sufficient incentives for farmers to invest in more profitable and cost-effective agricultural activities. The OFWM and HVA components focused on training for new technologies and improved on-farm water management to support a shift to higher-value crop production. The assumption was that trained farmers would transition to profitable commercial farming. This was expected to result in additional income from newly irrigated land, increase in high value-added crop cultivation, improved access to new markets, higher yields, and lower production costs.

There were several key assumptions underlying the OFWM and HVA farmer training program logic during the design of the investment:

  • The content and duration of training are sufficient to improve knowledge of targeted farmers, triggering behavior change.
  • Farmers and enterprises are appropriately selected to participate in the training.
  • Farmers have reliable irrigation water through existing structures or the Irrigation Infrastructure Activity financed by the compact.
  • Farmers have improved access to markets through the post-harvest, processing and marketing enterprise training component of the compact.
  • Farmers have access to credit for agricultural investments through existing finance mechanisms or the credit component of the compact.
  • Once irrigation, marketing and credit channels are sufficient, the primary barrier to farmer adoption of improved techniques and HVA is lack of knowledge.
  • Adoption of new, improved OFWM and HVA practices leads to an increase in farm productivity.
  • Increases in farm productivity lead to an increase in farm income that, in turn, leads to increases in overall household income. 

Measuring Results

MCC uses multiple sources to measure results. Monitoring data is used during compact implementation. Independent evaluations are generally completed post-compact. Monitoring data is typically generated by the program implementers and specifically covers the treatment group of farmers who received training under the compact. However, monitoring data is limited in that it cannot tell us what these farmers would have done in the absence of the MCC-financed training. For example, when implementers report that farmers have exceeded targets around the adoption of new techniques, we do not know if these farmers adopted because of the training or would have adopted without the training. This is why MCC invests in independent impact evaluations, which estimate a counterfactual to assess what would have happened in the absence of the investment.

The following table summarizes performance on output and outcome indicators specific to the evaluated activity:

 
Indicators Level Actual Achieved Target Percent Complete
Training/technical assistance provided for OFWM Output 45,639 45,000 101.4%
Training/technical assistance provided for HVA Output 36,070 36,000 100.2%
Number of farmers using better OFWM Outcome 26,424 28,834 91.6%
Number of farmers using better HVA practices Outcome 27,211 23,092 117.8%
Monitoring Indicators Tracked During Implementation of the Farmer Training Component of the Water to Market Activity

The average completion rate of output and outcome targets is 103 percent; and in three of the four indicators, targets were met or exceeded.

Evaluation Questions

The impact evaluation was designed to answer questions such as: 

  • Did the program affect the irrigation and agricultural practices of Armenian farmers?
  • Did the program affect agricultural productivity?
  • Did the program improve household well-being for the targeted farmers, including income and poverty? 

Evaluation Results

In the Armenia farmer training investment, although most output and outcome targets were met or exceeded, the independent evaluation did not detect impacts on adoption, productive income or household income. The results are summarized below, but it is the lessons learned, particularly around appropriate targeting of training participants and the design and implementation of complementary activities, that inform these findings. The OFWM and HVA farmer training was intended to complement several activities, including an irrigation infrastructure activity to increase reliable access to water. However, the irrigation infrastructure activity was significantly delayed during the compact period, and the farmer training implementation and evaluation continued before the infrastructure was complete. This was a fundamental break-down in the program logic.

 
Evaluator Mathematica Policy Research
Evaluation Type Impact
Methodology Randomized roll-out
Exposure Period 2-3 years for OFWM training; 1-2 years for HVA training
Adoption
  • No impacts detected on adoption of OFWM
  • Increase of 6 percentage points for soil preparation
  • Increase of 8 percentage points for purchase of pesticides from licensed store
  • No impacts detected on adoption of HVA
Farm Income
  • No impacts detected on farm investment
  • No impacts detected on productive income
Household Income
  • No impacts detected on rural poverty rate
  • No impacts detected on household income

Lessons Learned

MCC released impact evaluations from farmer training activities in five countries in October 2012. Looking across these five, and informed by lessons learned in impact evaluations in agriculture more broadly, MCC has identified a set of common lessons1. Several of the lessons, as illustrated by the Armenia case, are:

Next Steps

MCC has additional evaluations and analysis underway that will provide more results and learning about the Armenia Irrigated Agriculture Project:

  • Always return to the program logic. For integrated projects, it is especially important that the rollout of the farmer training is coordinated with complementary activities to maintain the program logic. Because the farmer training was not sequenced with the irrigation activity, or in some cases, not geographically linked, assumptions around farmers’ access to reliable water were not held. This potentially reduced the impact of the farmer training program on behavior change. In addition, other assumptions that farmers would have improved access to markets and to credit through existing or new structures did not hold during the evaluation period.
  • Balance ambitious targets with training effectiveness. Original targets were to train 60,000 farmers in OFWM and 300 enterprises with post-harvest, processing and marketing support. These targets were revised to 45,000 farmers and 225 enterprises as a result of irrigation re-scoping. However, the targets were still ambitious and may have resulted in a less effective approach to farmer training and participant selection. More targeted (and longer duration) trainings and technical support could be designed for different levels of farmers depending on their ability to adopt certain practices. In addition, the implementers’ focus on reaching training targets may have limited their attention to other program components.
  • The randomized roll-out evaluation approach has risks. In a randomized roll-out approach, a first round of treatment farmers is compared to a control group of farmers that receive training at a later date. The key to this approach is that there be enough time between the two phases to see behavior change and the accrual of benefits for the first farmers before the second round of farmers is trained. Timelines for farmer adoption of new practices, the five-year compact timeline and inevitable implementation delays can make a randomized roll-out a very risky approach. In the case of Armenia, the timing was such that the control group was trained before the irrigation infrastructure activity was completed, thereby losing the ability to compare between the two groups once irrigation was in place. Given the loss of the counterfactual, it is not possible to estimate the causal impact of the training on outcomes with the completed irrigation infrastructure.
    • Mathematica performance evaluation for Post-Harvest Processing Sub-Sub-Activity (December 2012)
    • Mathematica performance evaluation for Water to Market Credit Sub-Sub-Activity (December 2012)
    • Mathematica performance evaluation for Water to Market Institutional Strengthening Sub-Activity (December 2012)
    • Mathematica impact evaluation for Irrigation Infrastructure Activity that will assess the impacts of farmer training in combination with improvements in irrigation infrastructure (Spring 2015)
      • Through this evaluation, the independent evaluators will assess the impacts of farmer training in combination with improvements in irrigation infrastructure to answer the following questions: Is there an increase in the use of improved practices five to six years after training? Is there an increase in the use of improved practices with improved irrigation? If improved practices are being used, have they increased income?