Summary of Findings

Measuring Results of the El Salvador Production and Business Services Investments

View as PDF

In Context

The MCC compact with El Salvador was a five-year in- vestment (2006-2012) of $460.9 million in three projects: connectivity, human development and productive development. The Productive Development Project included three project activities implemented concurrently in the Northern Zone: (i) production and business services, (ii) investment support and (iii) financial services. The $55 million Production and Business Services Activity is the subject of both the results described here and an independent impact evaluation released by MCC in October 2012. This activity represents 12 percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.

Program Logic

The Productive Development Project was designed to transition producers to higher-profit activities, generate new investment, expand markets and sales, and create new jobs in ways that stimulate sustainable economic growth and poverty reduction. The Production and Business Services Activity included on-going technical assistance and training, in-kind donations (starter kits), demonstration plots, and technical and financial support for enterprises created and supported by the project in the dairy, horticulture and handicrafts value chains. 1

 

There were several key assumptions underlying the production and business services program logic:

  • Content and duration of training are sufficient to trigger behavior change.
  • Starter kits/in-kind donations are sufficient to trigger sustained behavior change.
  • Producers have necessary access to credit through existing structures supported by the Investment Support Activity or Financial Services Activity.
  • Primary barrier(s) to adoption of improved techniques is lack of knowledge and/or funds for investment.
  • Adoption of improved techniques leads to an increase in productivity
  • Increases in productivity lead to increases in productive income which, in turn, lead to an increase in overall household income.

 

Measuring Results

MCC uses multiple sources to measure results. Monitoring data is used during compact implementation. Independent evaluations are generally completed post-compact. Monitoring data is typically generated by the program implementers and specifically covers the treatment group of farmers who received training under the compact. However, monitoring data is limited in that it cannot tell us what these program participants would have done in the absence of the MCC-funded training. For example, when implementers report that participants have exceeded targets around the adoption of new techniques, we do not know if these participants adopted because of the training or would have adopted without the training. This is why MCC invests in independent impact evaluations, which estimate a counterfactual to assess what would have happened in the absence of the investment.

The following table summarizes performance on output and outcome indicators specific to the evaluated Transportation Project:

Monitoring Indicators Tracked During Implementation of the Production and Business Services Project

Indicators Level Actual Achieved Target Percent Complete
Number of enterprises assisted Output 272 292 93.2%
Number of farmers trained Output 15,363 10,465 146.8%
Number of beneficiaries of technical assistance and training – non-agriculture Output 2,104 3,035 69.3%
Number of hectares under production with support from the Productive Development Project Output 25,399 15,000 169.3%
Number of farmers who have applied improved techniques Outcome 11,520 7,000 164.6%
Number of enterprises that have applied improved techniques Outcome 164 114 143.9
 

 

The average completion rate of output and outcome targets is 131 percent; and in four of the six indicators, targets were met or exceeded.

Evaluation Questions

The impact evaluation was designed to answer questions such as: What was the impact of the Production and Business Services Activity in the dairy, horticulture and handicrafts value chains on the use of new practices, investment, employment creation, production, and income?

Evaluation Results

Although most output and outcome targets for the Production and Business Services Activity were met or exceeded, the independent evaluation found varied results for the three value chains. In dairy, the evaluation estimates there were impacts on adoption and increases in farm income. In horticulture, the evaluation estimates impacts on adoption, but no impacts on farm income. In handicrafts, the evaluation estimates impacts on employment for program participants, but no impacts were detected on productive income. In the horticulture evaluation, it should be noted that the sample was underpowered since only about 30 percent of the treatment group enrolled in the training program. This limits the ability to draw conclusions about ultimate impact, though the evaluation still provides ample opportunities for learning. In handicrafts, an interim evaluation was conducted, and the follow-up data will provide more information on whether or not the increase in employment led to an increase in productive or household income. Over the course of the compact, training design was modified. The evaluation results below capture the first phase of training that occurred from 2010 to2011. In handicrafts, additional evaluation work will assess the impact of the second phase.

 
Evaluator Mathematica Policy Research
Evaluation Type Impact
Methodology Randomized roll-out
Exposure Period 12 months
Adoption and employment For the 2010-2011 phase of dairy implementation:

  • 5 percentage points more likely to conduct quality control
  • 23 percentage points more likely to take measures to reduce costs
  • 7 percentage points more likely to report looking for new clients
  • 2 percentage points more likely to report selling to enterprises
  • .13 increase in annual employment generated by program participants (full-time equivalent jobs)
Farm Income For the 2010-2011 phase of dairy implementation:

  • $1,849 increase in net annual productive income

For the 2010-2011 phase of horticulture/handicrafts implementation:

  • No impacts detected on net annual productive income
Household Income For the 2010-2011 phase of dairy implementation:

  • No impacts detected on net annual household income or consumption by program participants

For the 2010-2011 phase of horticulture/handicrafts implementation:

  • No impacts detected on net annual household income or consumption by pro- gram participants

Lessons Learned

MCC released impact evaluations from farmer training activities in five countries in October 2012. Looking across these five, and informed by lessons about impact evaluations in agriculture more broadly, MCC has identified a set of common lessons. Several of the lessons as illustrated by the El Salvador case are:

  • Always return to the program logic. If the program logic and implementation plan include a variety of value chains, the evaluation must ensure sufficient power to track early and realistic impacts on income in each value chain. In El Salvador, the evaluation was not originally designed to be done by value chain but by all three sectors together. When unbundled, the design was “underpowered” to report on individual value chains, contributing to the uninformative evaluation of the horticulture value chain.
  • Linking to household income is difficult. In El Salvador dairy, the evaluators find that dairy farmers’ farm incomes roughly double that of the control group; however, they do not find an impact on household income or consumption. This is likely because the number of groups of dairy farmers that were randomized was small, and the evaluation was underpowered to report changes in household income by value chain. This needs to be taken into consideration for future evaluation design.
  • Test traditional assumptions. In El Salvador, some of the evaluation findings suggest that tailored trainings and donations may produce better results in the short-term. However, the evaluation was not designed to test effects of variation in training content or duration in order to confirm this. MCC and MCAs will look for future opportunities to use impact evaluations to test assumptions around the appropriate content and duration of training to maximize impact.
  • The randomized roll-out evaluation approach has risks. In a randomized roll-out approach, a first round of treatment farmers is compared to a control group of farmers that receive training at a later date. The key to this approach is that there be enough time between the two phases to see behavior change and accrual of benefits for the first farmers before the second round of farmers is trained. In the case of the handicrafts project, more will be learned with the follow-up data and impact analysis on intermediate and final outcomes. For the other value chains, however, the control groups have been trained as per the agreed roll-out methodology and additional learning using these evaluations is limited.

 

Next Steps

MCC has additional evaluations and analysis underway that will provide more results and learning about progress in El Salvador:

  • Mathematica final performance evaluation for Production and Business Services Activity (2013)
  • Mathematica final impact evaluation for the handicrafts value chain (2013)
  • Mathematica performance evaluation for Investment Support Activity (interim 2013 and final 2014)
Footnotes
  • 1. The evaluation covers dairy, horticulture and handicrafts. These are three of the eight value chains targeted in the investment, and the majority of the investment.