Measuring Results of the Georgia Agribusiness Development Activity

The $20 million Agribusiness Development Activity (ADA) was part of the Enterprise Development Project, and represented five (5) percent of the total compact. The ADA was designed to accelerate the transformation from subsistence to commercial agriculture by awarding resources to groups of farmers and enterprises motivated to apply innovative business solutions and technology to significantly boost household income and firm net revenue. The evaluation found that recipients reported increased investment, especially in agricultural equipment and machinery, and quantitative data confirmed  greater ownership of agricultural equipment and machinery. Both quantitative and qualitative data gave insight to intermediate outcomes, which suggested some impacts of the ADA, though not fully attributable to ADA due to data quality weaknesses. For instance, grantees employed more workers four years after application, as compared to pre-intervention levels. Grantees also experienced significant increases in their sales to wholesalers and were significantly less likely to experience business closure than non-grantees. However, by the end of the evaluation, quantitative and qualitative results indicated that both grantees and non-grantees experienced increases in wages, profits, and production.

Therefore, due to data limitations (e.g. high attrition rates), loss of a randomly selected control group, and insufficient statistical power, the Evaluator could not rigorously attribute any impacts to MCC’s investments.

In Context

The Government of Georgia (GoG) and Millennium Challenge Corporation (MCC) signed a $295.3 million Compact in 2005 that aimed to reduce poverty through economic growth by focusing on key constraints to development through rehabilitation of dilapidated infrastructure, improvements to roads and energy infrastructure, and investment  in SMEs and agribusinesses. An amendment to the Compact was signed on November 20, 2008 on provision of additional $100 million assistance. The compact was composed   of   two   projects:       the

Regional Infrastructure Rehabilitation Project and the Enterprise Development Project. The Enterprise Development Project included two activities, the Georgia Regional Development Fund and the Agribusiness Development Activity (ADA). At $20 million, the Agribusiness Development Activity represents five (5) percent of the total compact costs. This document describes the results of the Agribusiness Development Activity.

Program Logic

The Enterprise Development Project was designed to improve economic performance of agribusiness with the ADA grants designed to accelerate the transformation from subsistence to commercial agriculture. ADA awarded resources to groups of farmers and enterprises motivated to apply innovative business solutions and technology to significantly boost household income and firm net revenue. An application process required a sound business plan oriented to domestic market demand. Most awards were in the form of equipment or supplies matched with direct grantee investment in land, facilities, labor, training or additional equipment. There were several categories of grants:

  • Primary production and farmer partnerships – introduction of innovative agriculture production practices and partnerships between farmers (utilization of new genetics, and other inputs, grading,  packaging/branding for example) and farmer groups with processors or directly with the market (i.e. collection infrastructure, production contracts); 709 unique primary producers sought grants and 182 grants were awarded. $6,730,022 total funds were disbursed with $37,500 as the median grant;
  • Commercialization of service providers – introduction or expansion of retail outlets for the provision of seed, feed, fertilizer, genetics, equipment leasing through private sales and agronomic advice to farmers supporting technology transfer and productivity; 71 unique farm service centers (FSCs) applied for grants and 33 received them. $1,569,503 was disbursed to FSCs with a median grant of $49,500.
  • Machinery services – additional grants were made to farm service centers to enable greater mechanization of agricultural practices through renting or outsourced services of modern agricultural machinery; 12 FSCs initially qualified but only 10 grants were awarded. $1,499,250 was disbursed to FSCs with a median grant of $149,925.
  • Value-adding enterprises – transfer of technology to add value to raw product through quality assurance systems, collection, storage, distribution, value-added processing, packaging, advertising that positions final product competitively in the marketplace; 54 unique value adding enterprises (VAs) applied for ADA grants and 21 VAs received grants. $972,496 comprised total funds disbursed to VAs with a median grant of $49,700.
  • Value chain drivers – support to strategic enterprises whose services represent an important link in the value-chain such as wholesalers and aggregators who qualify for larger projects; 66  unique enterprises applied for these larger ADA grants and 37 were awarded. $5,027,187 comprised total funds disbursed to value-chain drivers with a median grant of $145,000.

The diagram below describes the program logic of ADA, highlighting the objective that ADA would increase production and efficiency of agribusinesses, thereby increasing revenues and decreasing production costs.

 

There were several key assumptions underlying the ADA program logic during the design of the investment:

  • Participants of the agriculture sector and food industry would incorporate business practices, market principals and modern technology into their operations.
  • Intensive consultation with stakeholders in Georgia’s food and agriculture industry, supported with sector and market assessments validated initial consensus that grantees would most likely represent a variety of value chains including horticulture, staple crops,  livestock (dairy and meat production and processing), and the service industry including input supply and leasing.
  • Cost-sharing was deemed an important mechanism to instill ownership and sustainable co- investment in enterprise development.
  • Close cooperation between ADA and the Georgian Regional Development Fund (GRDF) could identify possible opportunities for synergies or leveraging of investment funds.
  • Valid applications were expected to exceed funding capacities and thus some worthy projects would not be selected for implementation. This assumption justified randomization of qualified grantee selection to remove the opportunity for undue influence and subjectivity from the selection process.

 

Measuring Results

MCC uses multiple sources to measure results. Monitoring data is used during compact implementation. Independent evaluations are generally completed post-compact. Monitoring data is typically generated by the program implementers, and specifically covers the program participants who received treatment through the compact. However, monitoring data is limited in that it cannot tell us what these program participants would have done in the absence of the MCC-funded training. This is a key motivation for why MCC invests in independent impact evaluations, which estimate a counterfactual to assess what would have happened in the absence of the investment.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program, of which 3 of 5 key indicators were fully met or achieved during the life of the compact.

Indicators

Level

Achieved

(Sept 2011)

Target

Percent Complete

Jobs created 1

Output

3,264

3,377

97%

Firm Income 2

Outcome

$3,847,492

$1,046,000

368%

Household Net Income 3

Outcome

$12,306,272

$4,491,000

274%

Number of Direct Beneficiaries

Output

3,585

3,823

94%

Number of Indirect Beneficiaries

Output

103,772

72,173

144%

Evaluation Questions

The evaluation was designed to answer the following top-level question about the activity:

What is the ADA’s impact on economic growth and poverty reduction; including, to the extent data are fit for use, household incomes and expenditures, employment, production value (goods and services), value added, investments, imports, exports and other indicators in agriculture by country and region? What is the impact of the project on agricultural productivity and, if possible, income? 4

In addition, the following categories of outcomes were explored by the evaluation:

  • Net agribusiness revenue, number and type of employees, employee wages, and use of equipment that can be attributed to the ADA program

Evaluation Results

Evaluator

NORC at the University of Chicago

Methodology 5

  • Mixed methods approach that combined qualitative data with descriptive quantitative analysis
  • Qualitative data:
    • 11 focus group discussions (44 participants)
    • 25 in-depth interviews
  • Quantitative data:
    • Scored survey of ADA beneficiaries and a comparison group of applicants who did not receive ADA grants
    • Three survey rounds: 2008, 2010, and 2012.

Evaluation Period 6

2008-2012

Exposure Period

First grants disbursed in March 2007; Final grant disbursed December 2010.

Immediate Outcome

Recipients reported increased investment, especially in agricultural equipment and machinery, and quantitative data confirmed greater ownership of agricultural equipment and machinery.

Intermediate Outcome

Both quantitative and qualitative data gave insight to intermediate outcomes, with quantitative data suggesting some impacts of the ADA, though not fully attributable to ADA due to data quality weaknesses:

  • Quantitative survey results showed grantees employed more workers 4 years after application, as compared to those figures pre- intervention levels.
  • Grantees experienced significant increases in their sales to wholesalers.
  • Grantees were significantly less likely to experience business closure than non-grantees.

Focus groups and interviews also yielded interesting conclusions, while not necessarily robust as generalizable evaluation findings:

  • Grantees reported that they were able to diversify their businesses and increase their ability to provide a wider range of goods and services to their customers.

Evaluation findings also suggest that both grantees and non-grantees experienced some similar improvements in outcomes, such as improved access to and use of credit (loans).

Ultimate Impact

As quantitative and qualitative results indicated that both grantees and non-grantees experienced increases in wages, profits, and production, the Evaluator was unable to determine if the ADA led to these impacts or whether they were due to other intervening factors.

  • Due to data limitations (e.g. high attrition rates), loss of a randomly selected control group, and insufficient statistical power, the Evaluator could not rigorously attribute any impacts to MCC’s investments.

Lessons Learned

The lessons illustrated by the Georgia ADA Activity and its independent evaluation are:

Integrate implementers and evaluators early and align incentives.  The independent evaluator was brought in well after key program design actions had been taken and contracts terms and deliverables negotiated and signed with the implementer. In fact, the implementer’s contract made no reference to impact evaluation and the decision to use a lottery to allocate grants was imposed well after contract was signed, budgets were agreed and the implementer was mobilized and had developed a process for soliciting and selecting grant recipients. The management of both the intervention and its evaluation were both made much more challenging by not integrating design, implementation and evaluation planning decisions from the beginning.

 

  • Clearly articulate program logic. Program design must be sufficiently detailed to fully understand the process by which MCC’s inputs (e.g. ADA grants) lead to poverty reduction through economic growth. In other words, clear, detailed program logics for this type of grant program would help target beneficiaries and determine whether beneficiaries’ expected outcomes were actually realized.  While the early ADA documents listed a number of positive outcomes that were expected, there was no clear theory of change by which the activity could lead to these outcomes. In addition to not being clearly elaborated during project design, the logic also did not adapt as the project changed course during the process of implementation. Consequently, the linkage between different components of the activity were unclear and the impacts of the activities did not allow the evaluation to specifically compare expected outcomes to actual outcomes.

 

  • Careful judgment must be used in selecting programs and projects for rigorous impact evaluations. As described in the evaluation report’s Annex II, the ADA program was originally designed to foster a rigorous impact evaluation which was later impossible due to data limitations and implementation changes. MCC has two key takeaways from this evaluation’s failure to produce a rigorous measurement of impacts:
    • MCC should only select those programs for rigorous impact evaluations which have clearly measurable outcomes and a clear theory of change which maps out the expected realization of those outcomes.
    • Intensive mid-process technical reviews of impact evaluation design and implementation could detect and remedy these problems, a process MCC has already begun implementing across its evaluation portfolio in a format known as “Evaluation Management Committees.”

 

  • Successful and quality data collection requires collaboration between all stakeholders– including the Evaluator, MCA, both MCC and MCA M&E teams and data collection agencies and contractors. In terms of the data on which the evaluation relied, the household survey contained many flaws which limited its usability, and the sample size of households did not allow for generalizable results. Grantee and non-grantee survey data collection experienced many problems as well, such as delays in developing and procuring baseline data. This resulted in data that was intended for a baseline being implemented after some grantees had already received grants. Furthermore, much of the collected data was not usable, which highlights the important of a stronger, clearer role of evaluator oversight in data collection contracts. As the Evaluator was not a formal part of the data collection contract, they were left out of data collection fieldwork and encountered challenges in ensuring quality data. More collaboration between the Evaluator, the MCA, the M&E team, and other contractors (e.g. data collection agencies) could remedy this problem in future evaluations. In addition to the aforementioned Evaluation Management Committees, MCC has also begun standardizing contract language around Evaluators’ role in data collection oversight.

 

This was the first evaluation of an MCC grant-making program. Some lessons learned related to grant- making programs that come out of the ADA project 7  include:

  • Randomized lotteries are not just tools for impact evaluations.  In grant-making programs where demand (quality projects) significantly outpaces supply (resources), a randomized lottery can be an effective tool in addressing transparency and fairness issues in the selection process.
  • Cost-sharing by grantees requires enterprises to have a stake in the outcome of the grant and also leverages investment funds. The $16 million in ADA grants invested in the nearly 300 businesses leveraged more than $20 million in matching investment from grantees, making for a strong injection of capital to the sector. 8
  • Align application procedures with the level of sophistication of potential applicants. The project began with the assumption that there was excess of qualified demand. However, ADA implementers quickly learned that Georgian agribusinesses had difficulty successfully completing grant applications which resulted in poor quality of applications. Mid-course corrections were taken to simplify application forms, clarify evaluation criteria, target outreach, and provide guidance for completing applications. 9
  • Design processes and procedures for grant administration to mitigate fraud and abuse. For example, ADA introduced a milestone payment system whereby each grant was disbursed over time according to validation of the grantee achieving certain milestones associated with its cost-sharing schedule. Milestone payments mitigated the improper use of funds. However, note that greater risk mitigation measures require higher administrative costs. 10

 

As a result of the lessons learned in Georgia’s ADA, MCC project operational practices have changed in the following way:

 

  • Develop program logics early and revise as necessary. MCC now requires the formulation and revision of program logics from the concept note stage and throughout implementation. The program logic approach has been applied in the most recent cohort of compacts in development (Benin, Niger and Sierra Leone).  In addition, the agenda of MCC’s Ag College in Sep 2012 included a day devoted to review of program logic for all active agriculture projects in the portfolio by MCC and MCA counterparts together. This was followed up with a series of peer review discussions for each of the program logics to confirm links to on-going evaluations.
  • Assess grant programs critically. Careful consideration that grants making programs are appropriate and not market disruptive borne out of many and multiple discussions such as those that the MCC and MCG teams had when developing the process for selection and evaluation of the ADA program is now being more formally addressed at MCC.  In 2013, MCC started the process of developing guidance building on the ADA experience including the importance of establishing clear eligibility and selection criteria for awards and the necessity of performance monitoring capacity of grants management teams. In addition, in response to finding the right private-public balance, more recent grant making activities in MCC compacts such as Zambia   and Indonesia  also have co-funding requirements.

 

As a result of lessons learned during completion of the ADA evaluation and other recent evaluations, MCC evaluation management practices have changed in the following ways:

  • Formal review process for evaluations. The Monitoring and Evaluation unit has begun implementing a formal, cross-sectoral review process that defines critical milestones in the evaluation cycle that require substantive review and clearance by key internal stakeholders. This review process also requires local stakeholder review of key evaluation documents in consultation with the evaluator prior to submission to MCC in order to provide feedback on feasibility of proposed evaluation, as well as technical, and factual accuracy of evaluation documents. The formal review process is intended to ensure that evaluations are designed with stakeholder buy- in, using the program logic, appropriate methodologies for the timeframe of the expected results, and are flexible enough to adjust to changes in implementation.
  • Evaluation risk assessment. An Evaluation Risk Assessment Checklist has been developed and institutionalized by the Monitoring and Evaluation unit. The risk assessment checklist is reviewed by the M&E lead with M&E management. The risk assessment is intended to inform decision making and identify necessary course correction for more timely response to risk identification.
  • Development and use of standardized evaluation templates. The Monitoring and Evaluation unit has developed standardized templates in order to provide guidance internally and to independent evaluators on expectations related to evaluation activities and products. These templates are intended to clarify and raise standards for evaluations by influencing the daily work of M&E staff and evaluators. This is particularly relevant to the ADA evaluation, as descriptions of data collection oversight roles have been clarified and standardized in contract language rolled out to all new MCC independent evaluation contracts.

Next Steps

There are no next steps planned for this evaluation.

Footnotes
  • 1. Per the Georgia Compact M&E Plan (Rev. Dec 2009), this indicator was derived from grantee reports, as compiled by MCA Contractor CNFA. This number should not be considered as a rigorously attributable impact of the project.
  • 2. Ibid.
  • 3. Ibid.
  • 4. From Terms of Reference in NORC contract with MCC
  • 5. The ADA was initially structured to allow for an impact evaluation that incorporated a randomized design and regression discontinuity, however, the number of beneficiaries of the program proved too small and homogenous for a rigorous and useful randomized design. Problems plagued data collection as well, including issues with data quality assurance. Weak performance of the original data collection contractor rendered much of the early data unusable. Therefore, the impact evaluation design was converted into a performance evaluation which combined quantitative and qualitative data sources. The Evaluator provides insight into these and other challenges in Annex II to the Final Evaluation Report.
  • 6. All primary data collection was completed in 2012
  • 7. Lessons drawn from sources other than NORC’s independent evaluation report have sources cited in footnotes
  • 8. CNFA Final Report
  • 9. MCG Final Report on ADA Close-Out, S. Haile Trip Report 2007
  • 10. MCG Final Report on ADA Close-Out, CNFA Final Report