Measuring Results of the Honduras Farmer Training and Development Activity

Summary of Findings

In Context

The MCC compact with Honduras was a $205 million, five-year investment (2005-2010) in two projects:  (i) the Rural Development Project and (ii) Transport Project. The Rural Development Project included four activities: (i) farmer training and development, (ii) farmer access to credit, (iii) farm to market roads, and (iv) agricultural public goods grant facility. The Transport Project comprised three major activities, including the Highway CA-5 construction, Secondary Roads construction and rehabilitation, and a Weight Control System.

The $26.5 million Farmer Training and Development Activity (FTDA) is the subject of both the results described here and an independent evaluation released by MCC originally in October 2012. At the request of the independent evaluator, the original report was revised and the final version was posted in May 2014. This activity represents 13 percent of the total compact investment. The Transport Project and Farm to Market Roads Sub-Activity are the subject of forthcoming independent evaluations.

Program Logic

The Rural Development Project sought to improve the business skills, productivity, market access, and risk management practices of producers who operate small- and medium-size farms. This aimed to result in higher incomes for the targeted farmers, their employees and their communities and strengthen the capacity of those enterprises servicing horticultural production and trade. FTDA included on-going training and technical assistance, including financial support and extension services in commercial horticulture production and marketing.

There were several key assumptions underlying the Farmer Training and Development program logic during the design of the investment:

  • There were clearly defined, replicable criteria for selecting farmers into the farmer training program
  • Farmer training and small grants to farms would increase business skills and agricultural capacity of farmers and input providers.
  • The key constraint for farmers was lack of knowledge and skills on production and marketing of high- value horticulture.
  • The content and duration of technical training assistance and small grants to farms were sufficient to trigger behavior change.
  • Farmers had necessary access to credit through existing structures or through the Farmer Access to Credit Activity financed through the compact.
  • Adoption of good agricultural practices leads to an increase in farm productivity.
  • Increases in farm productivity lead to increases in farm income, which in turn leads to increases in overall household income.

Measuring Results

MCC uses multiple sources to measure results. Monitoring data is used during compact implementation. Independent evaluations are generally completed post-compact. Monitoring data is typically generated by the program implementers and specifically covers the treatment group of farmers who received training under the compact. However, monitoring data is limited in that it cannot tell us what these farmers would have done in the absence of the MCC-financed training. For example, when implementers report that farmers have exceeded targets around the adoption of new techniques, we do not know if these farmers adopted them because of the training or would have adopted them without the training. This is why MCC invests in independent impact evaluations when feasible, which estimate a counterfactual to assess what would have happened in the absence of the investment.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated activity using compact monitoring data:

Indicators Level Actual Achieved Target Percent Complete
Total number of recruited farmers receiving technical assistance Output 7264 8255 88.0
Number of technical assistance visits to program farmers Output 158,945 61,600 258.0
# of program farmers harvesting high-value horticulture crops Outcome 6029 6000 100.5
# of hectares harvesting high-value horticulture crops Outcome 9287 8400 110.6
# of business plans prepared by program farmers with assistance from the implementing entity Outcome 16,119 6960 231.6
Total value of net sales (millions USD) Outcome 75.5 N/A  
Total value of gross sales (millions USD) Outcome 153.0 N/A  
Total value of gross export sales (millions USD) Outcome 54.6 N/A  
Total value of gross local sales (millions USD) Outcome 98.5 N/A  

The average completion rate of output and outcome targets is 158 percent, and the number of indicators where targets were met or exceeded is four out of five, and four had no targets set.

Independent Evaluation Questions

The independent evaluation was designed to answer the following questions:

  • Did the FTDA intervention increase cultivation of horticultural crops?
  • Did the FTDA intervention increase household income?
  • Did the FTDA intervention increase employment on farms?

Independent Evaluation Results

The evaluation design for this activity changed over the course of the evaluation due to problems faced during implementation. In its original conception, the independent evaluator, NORC, and MCA-Honduras planned to use a randomized experimental design involving randomized assignment of communities (aldeas) to treatment. Following a series of implementation problems, the final approach used was an econometric model that relied on a model-based approach to impact evaluation.

As raised in the evaluator report, one key assumption is that the causal models are correct. This is based on the assumption that all important unobserved variables affecting selection, such as proven ex-ante ability to grow horticultural crops, are time invariant (i.e., are constant between the two survey rounds).

Evaluator: NORC at the University of Chicago
Methodology Failed randomized control trial converted to econometric model
  • The stable unit treatment value assumption (SUTVA, no macro effects assumption, partial equilibrium assumption) is made. This means that the effect (potential outcomes) on one individual are not affected by potential changes in the treatment exposure of other individuals. This implies, for example, that the program is not so large that the outcomes are correlated (e.g., that farmers would produce such a large amount of horticultural crops that the market would collapse).
  • The causal models are correct. The key assumption here is that all important unobserved variables affecting selection are time invariant (i.e., are constant between the two survey rounds).
  • The program intervention represents a “forced change” in (experimental control of) the agricultural system in Honduras.
  • The half of the country treated before this evaluation began is similar to the half yet to be treated, with respect to relationships among the important causal variables represented in the causal model underlying the statistical analysis.
Evaluation Period 12-36 months
Cultivation of horticultural crops

Model-based approach estimated net income change from horticultural crops is on average $600 higher for program participants than for nonparticipants. Input expenditures on these crops increased far more than they did for basic crops, implying a higher level of activity in cultivation of high value crops among program farmers. The results suggest a corresponding decline among program farmers in income from basic crops, as might be expected with changing crop mix; however, this decline is not statistically significant.

However, the program also did not appear to have had a positive effect on the proportion of farmers growing horticultural crops. This could well be because the implementer primarily chose as program participants farmers who showed a proven ability to grow horticultural crops. It is likely that increments in income from horticultural crops came from increased production among farmers already growing horticultural crops and not from farmers who switched over for the first time.

Household Income Even though there was an increase in income from horticultural crops, the evaluator did not find a corresponding statistically significant increase in net household income or household expenditures/consumption, as might have been expected.

Lessons Learned

MCC released impact evaluations from farmer training activities in five countries in October 2012. Looking across these five, and informed by lessons about impact evaluations in agriculture more broadly, MCC has identified a set of common lessons 1 . The specific lessons illustrated by the Honduras FTDA Activity are:

  • Integrate implementers and evaluators early. One key lesson is that MCC brought in the independent evaluator after key program design and implementation actions had been taken which affected the feasibility of a rigorous impact evaluation.
  • Clearly define program participants. For any intervention, MCC and country counterparts must work toward having clearly defined program participants and eligibility criteria when necessary.  In Honduras, there was a mix of broad selection criteria from MCA-Honduras and more specific selection criteria by the implementer in order to target farmers in the field. These two criteria although somewhat complementary, still resulted in challenges for replicating farmer selection for the purpose of a rigorous impact evaluation.
  • Align incentives. It is almost impossible to have a successful evaluation if program implementers and evaluators are not working in lock-step. This requires not only early integration, but also aligning incentives between the two. There must be clear understanding and commitment by the implementing entity to cooperate with the evaluator and vice versa. In Honduras, the implementer was contracted two years before the evaluator, which resulted in the implementer’s contract not including specific responsibilities regarding collaboration with the evaluator. In addition, the implementer was committed to delivering training to 6,000 farmers and increasing average income by $2,000. Therefore, the implementer’s incentive was to find successful program participants that were selected in part based on difficult to replicate criteria, which did not align with the evaluation design.

As a result of these lessons learned in Honduras in combination with lessons learned in four other farmer training evaluations, MCC project operational practices have changed in the following way:

  • Integrate implementers and evaluators early. Since 2011, MCC has worked to integrate evaluators as early in the project development as possible (El Salvador II, Georgia II, Indonesia, Malawi).
  • More carefully align incentives. For the new cohort of compacts, MCC it is ensuring that contractual requirements of implementers include the right incentives in terms of results for the intervention but also collaboration with and facilitation of impact evaluations.  

Based on the lessons learned from four other farmer training impact evaluations, MCC has also incorporated the following operational practices:

  • Develop program logics early and revise as necessary. MCC now requires the formulation and revision of program logics from the concept note stage and throughout implementation.  The program logic approach has been applied in the most recent cohort of compacts in development (Benin, Niger and Sierra Leone).   In addition the agenda of MCC’s Ag College in September 2012 included a day devoted to review of program logics for all active agriculture projects in the portfolio by MCC and MCA counterparts together.  This was followed up with a series of peer review discussions for each of the program logics to confirm links to on-going evaluations.
  • Assess training and technical assistance programs critically. Mixed results on adoption have led the MCC’s Agriculture Practice group to re-examine the focus on farmer training as a main part of the solution to low productivity of the agriculture sector and has resulted in more concerted efforts to identify interventions across the value chain.  If farmer training is considered, the duration, intensity and content of the training are more carefully examined and the benefits and challenges of reaching large numbers of beneficiaries are fully assessed. Equally important the use of grants and starter kits has led to a review of practices across all compacts and to the development of new guidance.

In addition, as a result of these lessons learned, MCC evaluation practices have changed in the following way:

  • Formal review process for evaluations. The Monitoring and Evaluation unit is pilot testing a formal review process that defines critical milestones in the evaluation cycle that require substantive review and clearance by key internal stakeholders. This review process also requires local stakeholder review of key evaluation documents in consultation with the evaluator prior to submission to MCC in order to provide feedback on feasibility of proposed evaluation, as well as technical, and factual accuracy of evaluation documents. The formal review process is intended to ensure that evaluations are designed with stakeholder buy-in, using the program logic, appropriate methodologies for the timeframe of the expected results, and enough flexibility to adjust to changes in implementation.
  • Evaluation risk assessment. An Evaluation Risk Assessment Checklist has been developed and institutionalized by the Monitoring and Evaluation unit. The risk assessment checklist is reviewed by the M&E lead with M&E management. The risk assessment is intended to inform decision making and identify necessary course correction for more timely response to risk identification.
  • Development and use of standardized evaluation templates. The Monitoring and Evaluation unit has developed standardized templates in order to provide guidance internally and to independent evaluators on expectations related to evaluation activities and products. These templates are intended to clarify and raise standards for evaluations by influencing the daily work of M&E staff and evaluators.

Next Steps

MCC has no immediate plans for further evaluation work on the Honduras Farmer Training and Development Activity.