Summary: Measuring Results of the Nicaragua Rural Business Development Project

In Context

The MCC compact with Nicaragua was a five-year investment (2006-2011) of $112.7 million in three projects: transportation, property regularization and rural business development 1 . The Rural Business Development Project included three activities: (i) rural business development services activity, (ii) technical and financial assistance activity, and (iii) improve farming and forestry activity. The $25.5 million combined rural business development services and technical and financial assistance activities are the subject of both the results described here and an independent impact evaluation released by MCC in October 2012. Together these activities reflect 22 percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.

Program Logic

The Rural Business Development Project was designed to increase the value added and productivity of farms and rural businesses in the departments of León and Chinandega by providing technical and financial assistance to farmers and rural business owners. Technical assistance consisted of improved access to technologies and improved access to markets. Financial assistance consisted of providing agricultural inputs and, in certain crops, entailed establishing rotating funds at the cooperative level. The program targeted four value chains: livestock, agricultural, non-agricultural, and forestry. The evaluation only directly covers livestock and agriculture, and within agriculture, covers the following crops: sesame, beans, vegetables, and cassava.

 

There were several key assumptions underlying the program logic of the Rural Business Development Services and Technical and Financial Assistance activities at the design stage:

  • Increase in farmer knowledge will lead to increased business skills and improved access to markets.
  • Content and duration of training are sufficient to generate behavior change.
  • A key barrier to farmer adoption of improved techniques is lack of knowledge.
  • Provision of in-kind donations, such as seed variety, fertilizer, irrigation equipment, stainless steel milk cooling tanks, and construction materials for on-farm milking stations, are sufficient to facilitate behavior change.
  • When newly acquired knowledge on improved techniques is adopted, they lead to increases in farm productivity.
  • Increases in productivity leads to an increase in farm income which, in turn, leads to increases in overall household income and an increase in farmer investment in farm activities.

 

Measuring Results

MCC uses multiple sources to measure results. Monitoring data is used during compact implementation. Independent evaluations are generally completed post-compact. Monitoring data is typically generated by the program implementers and specifically covers the treatment group of farmers who received training under the compact. However, monitoring data is limited in that it cannot tell us what these farmers would have done in the absence of the MCC-financed training. For example, when implementers report that farmers have exceeded targets around the adoption of new techniques, we do not know if these farmers adopted because of the training or would have adopted without the training. In the case of Nicaragua, the team was not able to set activity-level targets for these indicators, which made performance monitoring separately for each activity difficult 2 . To complement monitoring results, MCC invests in independent impact evaluations, which estimate a counterfactual to assess what would have happened in the absence of the investment.

The following table summarizes performance on output and outcome indicators specific to the evaluated activities:

Monitoring Indicators Tracked During Implementation of the Rural Business Development Project

Indicators Level Actual Achieved Target Percent Complete
Number of beneficiaries with business plans prepared with assistance from the Rural Business Development Project Output 7,881 N/A  
Number of beneficiaries implementing business plans Outcome 6,476 5,774 112.2%
Numbers of hectares, by sector, harvesting higher-value crops Outcome 18,037 N/A  
Number of jobs created Outcome 6,614 N/A  
Value of the beneficiaries’ investment Outcome $23,622,683 N/A  
 

 

The average completion rate of output and outcome targets is 112 percent. The number of indicators where targets were met or exceeded is one of one, and four did not have targets set.

 

Evaluation Questions

The impact evaluation was designed to answer the following questions:

  • Did farmers increase their use of improved technology?
  • Did farmers increase their farm income?
  • Did household income for farmers served by the business services improve?

 

Evaluation Results

The independent evaluation found varied results looking at farm income and household consumption. In addition, although the evaluation was not originally designed to test whether or not farm investments increased as a result of the training and increase in farm income, the evaluators did look at changes in investments in mobile and fixed capital in order to potentially explain why they were not finding changes in household consumption. Using traditional analysis methods, the evaluator found that farmers receiving technical and financial assistance experienced an estimated 15 percent increase in targeted farm income over the baseline level, but no impacts on mobile (e.g., tools and equipment but excluding livestock) or fixed capital (e.g., buildings, installations and fences located on the farmers’ property). Using a more innovative methodology that compares farmers with a shorter duration of exposure to treatment to farmers with longer exposure, the evaluator found that farmers with longer exposure experienced a 30 percent increase over their baseline level in targeted farm income, with a $2,000 increase in mobile capital and a $1,300 increase in fixed capital. The evaluation was not designed to measure impacts on immediate outcomes at the crop level, such as adoption of new techniques, which limits the evaluator’s ability to interpret results and MCC to learn what pathways are driving impacts on intermediate outcomes. The evaluator did not detect any impacts on household consumption, the proxy used for household income.

 
Evaluator Principle Investigators: Inter-Agency Agreement w/USAID; key evaluator: Michael Carter (University of California, Davis)
Evaluation Type Impact
Methodology Randomized roll-out
Exposure Period
  • Traditional methodology (binary treatment) – variable observation periods up to 24 months
  • Innovative methodology (continuous treatment) – variable observation periods up to 50 months
Adoption N/A at the crop level
Farm Income

Traditional methodology

  • 15 percent over baseline level of targeted farm income
  • No impacts detected on mobile or fixed capital

Traditional methodology

  • 30 percent increase over baseline level of targeted farm income
  • $2,000 increase in mobile capital (e.g., tools and equipment but excluding livestock
  • $1,300 increase in fixed capital (e.g., buildings, installations and fences located on the farmers’ property)
Household Income No impacts detected on household consumption (proxy for household income)

Lessons Learned

MCC released impact evaluations from farmer training activities in five countries in October 2012. Looking across these five, and informed by lessons about impact evaluations in agriculture more broadly, MCC has identified a set of common lessons 3 . Several of the lessons as illustrated by the Nicaragua case are:

  • Structure evaluation for learning. In Nicaragua, the evaluation was not designed to measure and understand the pathways by which changes in farm income occur. In addition, the original assumptions of the program logic did not question the content or duration of training, or the content of technical support to program participants, so these basic questions were not built into the evaluation design. This limited MCC’s ability to learn why changes in farm income are occurring, even though the findings of the evaluation suggest that the type of in-kind financial support and duration of technical services make a difference in adoption over time. In the future, MCC and MCAs will look for opportunities to use impact evaluations to test traditional assumptions about what works and structure evaluations for learning.
  • Linking to household income is difficult. In Nicaragua, the evaluators find that farm incomes increased; however, they do not find an impact on household consumption (a proxy for household income). This highlights two main issues around the difficulty of making the link to household income: (i) household income from a variety of sources is difficult to measure and (ii) the assumption that an increase in farm income leads to an increase in overall household income does not always hold. This may be because in the short to medium term, as households adjust to increases in farm income, the overall household income stays the same or can decline as a result of substitution from other sources. These points will be taken into consideration for future evaluations.
  • Work in lock-step. Despite the many challenges, this evaluation is an example of how effective partnerships can be developed between implementers and evaluators and incentives can be aligned resulting in successful impact evaluations. Conditions for success in Nicaragua were evident early on, given the effective partnership between MCA-Nicaragua and the evaluator, facilitated by a strong and independent MCA monitoring and evaluation department. This case is and should be a model for other compacts and demonstrates that MCA is a significant part of the equation.

 

Next Steps

This evaluation report is part of ongoing analysis to better understand the evolution of impacts over time. Revisions to this evaluation will be produced and published by MCC as warranted.

In addition to the evaluation results available now, there are two other complementary evaluations for Nicaragua’s Rural Business Development Project:

  • Final performance evaluation for plantain component (2012)
  • Final performance evaluation for rice component (2012)

 

Footnotes
  • 1. Conditions leading up to, during and following municipal elections of November 2008 were inconsistent with MCC’s eligibility criteria. In June 2009, MCC’s Board terminated a portion of the compact, reducing compact funding from $175 million to $113.5 million. While this action terminated the Property Regularization Project, the Rural Business Development Project was not affected.
  • 2. At the time that targets were set, the entire range of crops had not been determined; therefore, targets had not been set at the activity level.
  • 3. Issue Brief: MCC’s First Impact Evaluations: Farmer Training in Five Countries. October 2012. http://www.mcc.gov/documents/reports/issuebrief-2012002119501-ag-impact-evals.pdf

    Principles into Practice: Impact Evaluations of Agriculture Projects. October 2012. http://www.mcc.gov/documents/reports/paper-2012001116901-principles-impact-evaluations.pdf