Measuring Results of the Armenia Water-to-Market Activity

In Context

The $177.7 million Armenia Compact consisted of two projects: the Irrigated Agriculture Project and the Rural Road Rehabilitation Project. The Irrigated Agriculture Project includes two major activities— the Irrigation Infrastructure Activity and the Water- to-Market Activity (WTM)—and is equivalent to 86 percent of the overall compact investment. The subject of the evaluations summarized here is the $32.2 million WTM Activity, which includes four components: (1) the On-Farm Water Management (OFWM) and High-Value Agriculture (HVA) Farmer Training Sub-Sub-Activity, (2) the Credit Sub-Sub-Activity, (3) the Post-Harvest, Processing and Marketing (PPM) Sub-Sub-Activity, and (4) the Institutional Strengthening of Irrigation Management Entities Sub-Activity (ISSA). The Water-to-Market Activity represents 21 percent of the overall Irrigated Agriculture Project investment and 18 percent of the total compact.

Program Logic

The Irrigated Agriculture Project was designed to address the physical, managerial and financial investments needed to generate sustainable increases in rural incomes through irrigated agriculture. The WTM Activity was designed to complement the infrastructure investment by preparing and encouraging farmers to shift production to higher-value crop production, improving the capacity of the water-user associations (WUAs) to manage water resources, providing technical assistance to create improved marketing opportunities, and increasing access to credit for investment. The key underlying assumption of the WTM Activity was that investing in these separate areas simultaneously would create sufficient knowledge and incentives for farmers to take advantage of improved access to water and transition to more profitable and cost-effective agricultural activities. However, given implementation issues, it has been challenging to fully test this assumption as described below.

The program logic for the Irrigated Agriculture Project and its WTM Activity assumed that when farmers had reliable access to irrigated water, were trained on water management and productivity techniques and had access to credit, stronger post-harvest and marketing channels, improved irrigation delivery, and administrative management services provided by water-user associations, they would apply techniques and investment that transitioned their farm operations from low-value to high-value agricultural production. This was expected to result in additional farm income from increased agricultural productivity and market access, which was assumed to lead to an increase in overall household income.

 

 

There were several key assumptions underlying the Irrigated Agriculture Project and its WTM Activity program logic during the design of the investment:

  • Farmers have reliable irrigation water through existing structures or the Irrigation Infrastructure Activity.
  • Content and duration of training and technical assistance is sufficient to improve knowledge of targeted farmers and enterprises, triggering behavior change due to education levels of farmers.
  • Appropriate selection is conducted of farmers and enterprises to participate in the training or technical assistance.
  • Credit offered through the Credit Sub-Sub-Activity will be provided for a small percentage of trained farmers and will catalyze increased agriculture lending through existing finance mechanisms and enable farm investments.
  • Access to markets facilitated through existing mechanisms or the Post-Harvest, Processing and Marketing Sub-Sub-Activity will be sufficient.
  • Adoption of new, improved agricultural practices leads to an increase in farm productivity.
  • Increase in farm productivity, along with improved marketing channels, lead to increase in farm income.
  • Increase in farm income leads to increase in overall household income.

During implementation, the various project activities were disjointed in targeting project beneficiaries and sequencing. The irrigation infrastructure rehabilitation activities were significantly reduced in scope and faced construction delays of two to three years. Despite irrigation construction scope changes and delays, farmer training in improved technologies commenced for farmers that already had some access to reliable water and/or would benefit from irrigation rehabilitation efforts. The estimates of reliable access to water for farmers trained ahead of the irrigation rehabilitation proved to be inaccurate for some communities. Credit on-lending and training of financial institutions began approximately at the same time as training efforts, and, by design, limited in scope and resources reaching less than 2% of trained farmers. Post-harvest, processing and marketing activities scaled up activities approximately two years after farmer training began and completed major initiatives, such as creation of consolidation centers, in the final year of implementation. Finally, water user association capacity building occurred alongside other project activities with strong complementarities with the on-farm water management training content, although actual access to water varied and infrastructure rehabilitation also was completed only in the final year of the compact.

Measuring Results

MCC uses multiple sources to measure results, including monitoring data during compact implementation and independent evaluations, which in many cases are continued after compact closeout. Monitoring data is typically generated by the implementers, and specifically covers the group of farmers who received training under the compact.

The table below includes the monitoring indicators that were tracked during implementation of the WTM Activity.

Monitoring Indicators Tracked During Implementation of the Water to Market Activity

Indicators Level Actual Achieved Target Percent Complete 1
Farmer Training
Number of farmers using better on-farm water management Outcome 26,424 28,834 91.6%
Number of farmers using better high-value agriculture practices Outcome 27,211 23,092 117.8%
Training/technical assistance provided for on-farm water management (number of farmers) Output 45,639 45,000 101.4%
Training/technical assistance provided for HVA (number of farmers) Output 36,070 36,000 100.2%

 

Credit

Number of loan borrowers Output 1,008 N/A N/A
Loans provideds Output $13,133,200 $8,500,000 154.5%

 

Post-Harvest, Processing and Marketing

Number of enterprises that have applied improved techniques Outcome 178 112 159%
Training/technical assistance provided for postharvest, processing and marketing services (number of enterprises) Output 227 225 100.9%

 

Institutional Strengthening of Irrigation Management Entities 2

Recovery of water-user association operations and maintenance cost by water charges Outcome 48.1 60 48.9% 3
Increased collection of irrigation service fee for the water used Outcome 68.7 55 374% 4
Percentage of WUA disputes resolved by the Dispute Resolution Committee for the eight targeted WUAs Outcome 9.8 15 65.4%
Management improvement plans developed for WUAs Output 44 44 100%
 

 

The average completion rate of output and outcome targets is 129 percent, and the number of indicators where targets were met or exceeded is seven of 11. It should be noted that these numbers are not always the same as the evaluation results because, in addition to not taking the “without project scenario” into account as described below, the monitoring data comes from different data sources, data collection instruments and samples of respondents.

Monitoring data is limited in that it cannot tell us what these farmers would have done in the absence of the MCC-funded training, credit or technical assistance. For example, when implementers report that farmers have exceeded targets around adoption of new techniques, we do not know if these farmers adopted because of the training or would have adopted without the training. This is a key motivation for why MCC invests in independent impact evaluations, which estimate a counterfactual—what would have happened in the absence of the investment. For some activities, impact evaluations are not feasible or cost-effective and in those cases, MCC invests in independent performance evaluations. The evaluations for the WTM Activity combine the use of impact evaluations and performance evaluations.

Summary of Water-to-Market Activity Evaluations

Summary of Water-to-Market Activity Evaluations
Component Evaluation Type Methodology
Farmer training Impact Randomized roll-out
Credit Performance Matched comparison group
Post-Harvest, Processing and Marketing Performance Ex-post
Institutional Strengthening of Irrigation Management Entities Performance Pre-post
 

Evaluation Questions

The evaluations of each component of the WTM Activity were designed to answer questions such as:

  • How was the component implemented? What were the characteristics of each component’s participants, and how were these participants identified and recruited? What assistance was provided to participants through the component?
  • What were the impacts or potential contributions to changes in outcomes of the component? What were the components’ impacts or potential contributions to changes in the use of new practices or technologies? What were the impacts or potential contributions on key outcomes such as household income and poverty?

 

Evaluation Results

WTM Activity Overall

Because the evaluations of WTM credit, ISSA and PPM were introduced after the WTM training was already underway, it was not possible to design a quantitative evaluation that could rigorously examine the overall effects of the combined WTM Activity. However, the evaluations have attempted to gauge the magnitude of the possible overall effect of WTM by considering the evidence available from the evaluations of each of the four components. The WTM components were not well integrated with each other, so there is little chance that the planned complementarities were realized. For this reason, when assessing the overall effect of WTM, the evaluation assesses the possible effect of each component on its target population. WTM training was the largest and most visible component, but it had little impact on the overall WTM goals of increasing agricultural production, agricultural profits and household income. Thus, any overall effects of WTM could only be through direct effects of the other components. There is suggestive evidence that WTM credit and PPM may have had effects on the beneficiaries who participated in these components, but little evidence to suggest that these components had broader effects beyond the direct beneficiaries. Although some participants may have benefited from these components, the overall effect of WTM on the full set of targeted beneficiaries was probably small, at least as of the end of the compact. However, many of the potential effects of ISSA on farmers were designed to provide benefits beyond the compact period in the form of sustained irrigation infrastructure investments and more effective WUAs; if ISSA is successful in these goals, it would affect many farmers. A final evaluation of the irrigation infrastructure improvements is planned for 2015, which will also follow-up on the effectiveness of ISSA in preparing WUAs to maintain the irrigation infrastructure.

Farmer Training

In the OFWM and HVA farmer training component, although the average completion rate of output and outcome targets reported by monitoring data was 103 percent, the independent impact evaluation did not detect impacts on adoption of on-farm water management practices, transition to high-value crops, productive income or household income. Some impacts were detected on simple high-value practices such as soil preparation and purchase of pesticides from a licensed store. The results are summarized below, but it is the lessons, particularly around appropriate targeting of training participants and design of complementary activities, such as access to credit and infrastructure, that inform these findings. The OFWM and HVA farmer training was intended to complement several activities and sub-activities, including an irrigation infrastructure activity to increase reliable access to water. However, the irrigation infrastructure activity was significantly delayed during the compact period, and the farmer training implementation and evaluation continued without the new and improved infrastructure (a projected amount of 9,000 new hectares under irrigation and 38,000 hectares of improved irrigation). This was a fundamental breakdown in the program logic and a key lesson learned for MCC currently being applied in similar ongoing compact activities.

Farmer Training
Evaluator Mathematica Policy Research
Evaluation Type Impact
Methodology Randomized roll-out
Exposure Period 2-3 years for OFWM training; 1-2 years for HVA training
Adoption
  • No impacts detected on adoption of OFWM
  • Positive impacts on the adoption of some HVA practices
    • Increase of 6 percentage points for soil preparation ase of 8 percentage points for purchase of pesticides from licensed store
  • No impacts detected on adoption of HVA crops
Farm Income
  • No impacts detected on farm investment
  • No impacts detected on productive income
Household Income
  • No impacts detected on rural poverty rate
  • No impacts detected on household income

Credit

In the WTM credit component, the independent performance evaluation detected potential effects on adoption, crop production and household income. WTM credit recipients were more likely to make agricultural investments, had higher production and realized larger incomes. However, the evaluation has some important limitations and the results must be viewed in the context of those limitations. First, the evaluation uses a matched comparison group which cannot control for unobserved differences between the treatment and comparison groups (such as motivation). This means that the results are likely upward biased. Second, the evaluation was conducted on a very small sample size, which means that the true effects of the program may not be well-measured. The results are summarized below, which are presented as potential effects, but not conclusive impacts.

Credit
Evaluator Mathematica Policy Research
Evaluation Type Performance
Methodology Matched comparison group
Exposure Period 1-3 years
Adoption
  • Increase of 15 percentage points for establishing a greenhouse
  • No impacts detected on establishing or renewing an orchard
Farm Income
  • $1,946 increase in net annual productive income
Household Income
  • $2,333 increase in net annual household income
  • No impact detected on household consumption

Post-Harvest, Processing and Marketing

In the Post-Harvest, Processing, and Marketing component, the independent performance evaluation provides suggestive information about the effect of the technical assistance and training provided to enterprises on self-reported business outcomes and perceptions. The majority of recipients of PPM assistance reported positive outcomes, including improved product and service quality, increased productivity and increased sales. However, the evaluation has some important limitations and the results must be viewed in the context of those limitations. All results are ex-post results reported by assisted enterprises. The results do not take into account what would have happened without the assistance nor do they take into account any baseline information. The results are summarized below, which are presented as suggestive effects, but not conclusive.

Post-Harvest, Processing and Marketing
Evaluator Mathematica Policy Research
Evaluation Type Performance
Methodology Ex-post
Exposure Period 1-3 years
Evaluation Findings
  • 84 percent reported improved product and service quality
  • 69 percent reported increased productivity
  • 67 percent reported increased sales
  • 78 percent who reported improved product or service quality reported that PPM greatly or somewhat supported this
  • At leas t 70 per cent who r eported incr eased sales, incr eased pr oductivity, lo wer pr oduction c osts, and increased profits attributed these at least in part to PPM

Institutional Strengthening of Irrigation Management Entities

In the Institutional Strengthening of Irrigation Management Entities component, the independent performance evaluation provides suggestive information about the effect of the strengthening activity on water-user association sustainability. The evaluation reports that membership rates and membership fee payment rates increased moderately during implementation. In addition, WUAs improved their financial standing, but they are not yet approaching financial self-sufficiency. These improvements cannot be attributed fully to the component, because many other factors affect these outcomes, such as weather and national policies. The results are summarized below, which are presented as suggestive effects, but not conclusive.

Institutional Strengthening of Irrigation Management Entities
Evaluator Mathematica Policy Research
Evaluation Type Performance
Methodology Pre-post
Exposure Period 1-2 years
Evaluation Findings
  • Increase in payment of membership fee from 75 to 92 percent of members
  • Knowledge of having a village representative in the WUA increased from 27 to 52 percent
  • Increase in annual water payments from $76 to $98 per user, although no increase in the proportion of users making payments
  • Increase in cost recovery from 37 to 48 percent
  • Irrigation policy efforts may result in changes in the sustainability of the irrigation sector

Lessons Learned

MCC released impact evaluations from farmer training activities in five countries in October 2012. Looking across these five, and informed by lessons from impact evaluations in agriculture more broadly, MCC has identified a set of common lessons. 5 Several of the lessons are illustrated by the Armenia case. Two additional lessons learned from reviewing all four Armenia evaluations were also identified.

  • Always return to the program logic. It is especially important in integrated projects that the rollout is coordinated with complementary activities. In the case of Irrigated Agriculture Project, this means the coordination of the farmer training rollout with the irrigation infrastructure activity and post-harvest marketing and access to credit components. Because the farmer training was not sequenced with the irrigation activity or completely geographically linked, assumptions around farmers’ access to reliable water were not held, potentially reducing the impact of the farmer training program on behavior change. In addition, other assumptions around importance of improved access to markets (post-harvest, processing and marketing component) and access to credit through existing or new structures did not hold during the evaluation period either. Farmer training began without discreet links to buyers—marketing efforts were delayed until the third year of the compact, and the Access to Credit Sub-Sub Activity served only a small number of farmers and did not succeed in promoting additional lending to the agriculture sector.
  • Balance ambitious targets with training effectiveness. Original targets were to train 60,000 farmers in on-farm water management, 30,000 in high-value agriculture practices and 300 enterprises with postharvest, processing and marketing support. These targets were revised to 45,000 farmers for on-farm water management training, 36,000 farmers for high-value agriculture training and 225 enterprises as a result of lessons learned during implementation, a smaller scope of the irrigation rehabilitation, currency devaluation, and the difficulty in finding 60,000 appropriate farmers and 300 enterprises to participate in training. However, the targets were still ambitious and might have resulted in a less-effective approach to farmer training, selection of participants and limited attention to the post-harvest, processing and marketing activities. More targeted and longer duration of trainings and technical support could be designed for different levels of farmers depending on their ability to adopt certain practices. The structure of the lump-sum contract with the implementer also drove the sequencing, whereby they were compensated based on meeting training targets rather being rewarded for changes in program participants’ income.
  • The randomized roll-out evaluation approach has risks. For the farmer training impact evaluation, the evaluators used a randomized roll-out approach in which a first round of treatment farmers is compared to a control group of farmers that received training at a later date. The key to this approach is that there is enough time between the two phases to see behavior change and the accrual of benefits for the first farmers before the second round of farmers is trained. Timelines for farmer adoption of new practices, the five-year compact timeline and inevitable implementation delays made the randomized roll-out a risky approach. In the case of Armenia, the timing was such that the on-farm water management and highvalue agriculture control group was trained before the Irrigation Infrastructure Activity was completed, thereby losing the ability to compare between the two groups once irrigation was in place. Given the loss of the counterfactual, it is not possible to estimate the causal impact of the training on outcomes with the completed irrigation infrastructure or even to allow for more crop cycles and an adjusted (more realistic) timeline for behavior change. This is a potential risk that should be considered for future impact evaluations using a randomized roll-out methodology.
  • A multifaceted development approach requires proactive and visionary management. The Irrigated Agriculture Project suffered from poor integration of project activities and targeting of beneficiaries. Project activities were broken into several different contracts, which increased the challenge of coordination among contractors’ timelines and activities. Mid-course corrections such as improvements in coordination among contractors, implementation strategy and staffing changes reduced the risks inherent in the piecemeal implementation approach, which improved implementation performance. Nonetheless, sequencing challenges compromised the original program logic.
  • The evaluation questions are based on the program logic and must be designed carefully from the beginning to understand the scope and limitations of the evaluation. Given that the WTM Activity was not designed and implemented as a package of coordinated interventions for a targeted group of beneficiaries, MCC could not design an evaluation of the overall WTM Activity. The project design, implementation and the corresponding independent evaluations have limited MCC’s ability to report on the overall impact of the WTM Activity. In the future, MCC should work with all stakeholders to understand the program logic, how the program will be implemented and clarify what the evaluation will be able to answer and not answer from the beginning.

 

Next Steps

MCC has an ongoing independent evaluation of the Irrigation Infrastructure Activity that will provide more results and learning about the Irrigated Agriculture Project (Spring 2015).

Through this evaluation, the independent evaluators will try to assess the effects of farmer training in combination with improvements in irrigation infrastructure to answer the following questions: Is there an increase in the use of improved practices five to six years after training? Is there an increase in the use of improved practices with improved irrigation? To what extent did farmers shift to higher-value agriculture production? If improved practices are being used, have they increased income? If farmers shifted to higher value agriculture, did it increase their income? Are the irrigation infrastructure improvements expected to be sustained by the WUAs?

Footnotes
  • 1. Percent complete is calculated with the following formula: (actual - baseline)/(target - baseline).
  • 2. The actuals reported for the outcomes indicators for this component cover 2010; however the targets are for 2011. Data for 2011 was not available at the end of the Compact.
  • 3. The baseline for this indicator was 36.7% therefore, the percent complete is calculated as (48.1-36.7)/(60-36.7).
  • 4. The baseline for this indicator was 50% therefore, the percent complete is calculated as (68.7-50)/(55-50).
  • 5. Issue Brief: MCC’s First Impact Evaluations: Farmer Training in Five Countries. October 2012. http://www.mcc.gov/documents/reports/issuebrief-2012002119501-ag-impact-evals.pdf
    Principles into Practice: Impact Evaluations of Agriculture Projects. October 2012. http://www.mcc.gov/documents/reports/paper-2012001116901-principles-impact-evaluations.pdf