The MCC compact with Ghana was a five-year investment (2006-2011) of $547 million in three projects: transportation, rural development and agriculture. The agriculture project included six activities: (i) commercial training for farmer-based organizations (FBOs), (ii) irrigation, (iii) land, (iv) post-harvest, (v) access to credit, and (vi) feeder roads. The $62.5 million Commercial Training Activity is the subject of both the results described here and an independent impact evaluation released by MCC in October 2012. This activity represents 11 percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.
The Agriculture Project was designed to enhance the profitability of staple food and horticulture crops and to improve delivery of business and technical services to support the expansion of commercial agriculture among FBOs. FBOs are mainly groups of eligible farmers, in addition to input suppliers selling to such farmers or output processors buying from such farmers. The objective of the Commercial Training Activity was to accelerate the development of commercial skills and capacity among FBOs by providing training in management, business planning, technology applications, and marketing. To provide farmers with start-up capital and an incentive to participate in the training, each farmer was given a starter kit that contained fertilizer and seeds for one acre, protective equipment and a small amount of cash to facilitate land clearing. As a result of the training and starter kit, FBOs were expected to be more efficient in production, gain scale in purchasing inputs and competitively respond with necessary volume and quality to commercial demands. The program logic is summarized below.
There were several key assumptions underlying the Commercial Farmer Training program logic at the design stage:
- Content and duration of the training are sufficient to trigger behavior change.
- Content of the starter kits is sufficient to allow farmers to apply the methods learned in training.
- FBO members develop business plans during training, which may enable them to obtain credit through existing structures or the Access to Credit Activity.
- FBO members have sufficient water to generate increases in productivity through existing structures.
- Training will increase productivity because FBO members are constrained by their limited knowledge of agribusiness practices and technical skills.
- Starter kits will increase productivity because FBO members are constrained by access to quality inputs and the limited availability of investment capital.
- Increases in farm productivity lead to an increase in farm income which, in turn, leads to increases in overall household incomes.
MCC uses multiple sources to measure results. Monitoring data is used during compact implementation. Independent evaluations are generally completed post-compact. Monitoring data is typically generated by the program implementers and specifically covers the treatment group of farmers who received training under the compact. However, monitoring data is limited in that it cannot tell us what these farmers would have done in the absence of the MCC-financed training. For example, when implementers report that farmers have exceeded targets around the adoption of new techniques, we do not know if these farmers adopted because of the training or would have adopted without the training. This is why MCC invests in independent impact evaluations, which estimate a counterfactual to assess what would have happened in the absence of the investment.
The following tables summarize performance on outcome and output indicators specific to the evaluated activities:
|Indicators||Level||Actual Achieved||Target||Percent Complete|
|Number of FBOs trained in commercial agriculture||Output||1,242||1,000||124.2%|
|Number of farmers trained in commercial agriculture||Output||66,930||50,000||133.9%|
|Number of farmers adopting new technologies and farming methods||Outcome||59,060||42,500||139.0%|
|Number of enterprises that have applied improved techniques||Outcome||535||700||76.4%|
|Percent of FBOs meeting market targets||Outcome||97.21||60||162.0%|
|Hectares under production with MCC support||Outcome||58,568||53,060||110.4%|
|Percent of post-harvest loss at farm-gate: maize*||Outcome||11.31||10.3||48.6%|
|Percent of post-harvest loss at farm-gate: fresh pepper*||Outcome||5.05||4.5||31.3%|
|Monitoring Indicators Tracked During Implementation of the Commercial Training Activity|
*These two indicators report on percentage loss, which means that in order to meet or exceed the target, the percent must be equal to or lower than the target set.
The average completion rate of output and outcome targets is 103 percent; and in five of eight indicators, targets were met or exceeded.
The impact evaluation was designed to answer the following questions:
- Did the training and starter kit lead to increases in total land cultivated and labor hours for farm activities?
- Did the training and starter kit lead to the use of higher quality inputs, such as seeds and agrochemicals?
- Did the training and starter kit change the value or source of loans obtained by farmers?
- Did the training and starter kit increase crop income and yields?
Although most output and outcome targets for this activity were met or exceeded, the independent evaluation found varied results for the three regions invested in under the Commercial Training Activity. The evaluation showed no impact on yields or crop incomes on average across the three regions. However, northern region farmers’ annual crop income increased significantly relative to the control group, over and above any impacts recorded in the other zones. Southern region farmers’ annual crop income decreased significantly relative to the control group, beyond any impacts recorded in the other zones. There was no significant impact in the Afram zone. There are several factors that may contribute to these findings. First, due to delays in implementation, the time between the treatment group and control group receiving training was reduced to from two years to one year. As a result, the evaluation only captured impacts after one year, even though the original program logic assumed two crop cycles would be necessary to observe and measure a change in outcomes. In addition, training activities may not have been tailored appropriately to the regional differences in farmer capacity and crops.
|Evaluator||The Institute for Statistical, Social and Economic Research (ISSER) at the University of Ghana|
|Exposure Period||12 Months|
|Adoption and employment||
The evaluator reached the following conclusions (more can be found in Chapter 11 of the full report):
|Farm Income||In Ghana, northern region farmers’ annual crop income increased significantly relative to the control group, over and above any impacts recorded in the other zones|
MCC released impact evaluations from farmer training activities in five countries in October 2012. Looking across these five, and informed by lessons about impact evaluations in agriculture more broadly, MCC has identified a set of common lessons. 1 Several lessons as illustrated by the Ghana case are:
- Test traditional assumptions. The evaluation findings suggest that starter kits may not have served as an incentive for adopting fertilizer application and the use of improved seeds as expected. In addition, it appears that aggressive training targets might have impacted content and quality of training, including lack of customized training and starter kits to regions and priority value chains per region. The findings suggest the need to weigh carefully the costs/benefits of large scale, limited-duration training versus more focused, longer-duration training. The original assumptions of the program logic did not question the content or duration of training, or the content of technical support to program participants, so these basic questions were not built into the evaluation design. This has limited MCC’s ability to understand more about what did and didn’t work and why in a rigorous way. In the future, MCC and MCAs will look for opportunities to use impact evaluations to test traditional assumptions about what works and specifically will look for opportunities to use impact evaluations to test starter kit contents, delivery method and timing, and assess how they affect farming incentives and behaviors.
- Maintain realistic time horizons. Understanding and setting realistic time horizons, particularly in the context of changing implementation schedules, is crucial. In Ghana, only one crop cycle in the North was observed during the evaluation period; this may not be enough time to expect to see a change in overall farm and household income. MCC has learned that when projects change or are delayed during implementation, it is important to make deliberate decisions about whether to reschedule the surveys or adjust the methodology in order to maintain realistic time horizons to detect a measurable change in outcomes. In addition, MCC and MCA should use impact evaluation as a tool to test assumptions about appropriate time horizons since little is known on how much time is required to expect to see a change in outcomes as a result of farmer training. Impact evaluation can be used to test whether or not assumptions about time horizons for adopting new techniques hold true.
- The randomized roll-out evaluation approach has risks. In a randomized roll-out approach, a first round of treatment farmers is compared to a control group of farmers that receive training at a later date. The key to this approach is that there be enough time between the two phases to see behavior change and the accrual of benefits for the first farmers before the second round of farmers is trained. Timelines for farmer adoption of new practices, the five-year compact timeline and inevitable implementation delays can make a randomized roll-out a very risky approach. In the case of Ghana, the randomized roll-out resulted in training the control group before the end of the compact, allowing only one crop cycle between the first treatment and the incorporation of the controls. Given the loss of the counterfactual, it is not possible to estimate the causal impact of the training on outcomes with additional crop cycles.
- Large complex projects are a challenge to manage, implement and measure. In large, multi-faceted projects, it is important to define clearly the program’s intended causal pathway to growth for two reasons: to ensure that the program design is tailored adequately to local requirements and to ensure that the impact evaluation can measure the appropriate indicators and generate the needed learning. Ghana’s Agriculture Project and its six separate and complex activities made this a challenge for both implementers and evaluators. Implementation challenges may have diverted attention away from ensuring the impact evaluation was fully aligned with the content of the FBO training and customized per region to the priority crops and value chains.
MCC is committed to more analysis of the data and evaluation report to further assess regional differences for key indicators and develop a more comprehensive picture of the changes associated with the activity. In addition to the evaluation results available now, there are five other complementary evaluations for Ghana’s Agriculture Project:
- Final impact evaluation on Land Activity (2014)
- Final impact evaluation on Feeder Roads Activity (TBD)
- Final performance evaluation for Irrigation Activity (2015)
- Final performance evaluation for Post-Harvest Activity (2015)
- Final performance evaluation for Access to Credit Activity (2015)
1. Issue Brief: MCC’s First Impact Evaluations: Farmer Training in Five Countries. October 2012. http://www.mcc.gov/documents/reports/issuebrief-2012002119501-ag-impact-evals.pdf
Principles into Practice: Impact Evaluations of Agriculture Projects. October 2012. http://www.mcc.gov/documents/reports/paper-2012001116901-principles-impact-evaluations.pdf