Measuring Results of the El Salvador Productive Development Project

Create PDF

The $68 million Productive Development Project represented 15% of the El Salvador Compact and consisted of three activities designed to transition producers to higher-profit activities, generate new investment, expand markets and sales, and create new jobs: i) Production and Business Services, (ii) Investment Support, and (iii) Financial Services. The independent interim  impact evaluation on the Production and Business Services (PBS) Activity, covering the 2009-2010 phase of implementation, found varied results for the three value chains examined: impacts on employment for program participants, but no impacts on productive income in handicrafts; impacts on adoption, but no impacts on farm income in horticulture; and impacts on adoption and on farm income in dairy. Additional follow-up on the handicrafts sector showed that by the end of 2012 the increase in employment detected in the 2009-2010 phase of implementation had disappeared; no impacts were detected on net handicraft income, net annual household income, or household consumption; and negative impacts were detected on salaried income. The final data collection rounds for dairy and horticulture were cancelled due to lack of statistical power and changes in program implementation; therefore, the final evaluation of the dairy and horticulture sectors relied upon administrative data, which indicated that PBS assistance surpassed performance targets for increased production, employment, and sales. Because administrative data do not take into account what would have happened without the project, and the interim impact findings are not generalizable, it is impossible to make a definitive conclusion regarding the impact of the full PBS assistance package from 2008 to 2012.

 

As for the Investment Support Activity, although it fell short of its original lending targets, interviewed credit recipients appear to have experienced higher levels of investment, employment, production and sales than non-credit recipients. However, the comparisons of recipients and non-recipients merely suggest the potential positive effect of the credit on these key outcomes, as there is not a valid comparison group nor a sufficient sample size to attribute differences to the activity.

 

Under the Financial Services Activity, the two guarantee programs, PROGARA Norte and SGR, together surpassed program targets for number of guarantees provided and value of guarantees. Most recipients were microenterprises and defaults for both programs were low. According to implementer staff, these programs expanded their services to clientele that likely could not have found credit elsewhere. Technical assistance to financial institutions appeared to have mixed results and lacked clear performance metrics. Cultural issues and the complexity of insurance policies are believed to be the main reasons behind the failure of the crop insurance program.

In Context 1

The MCC Compact with El Salvador was a five-year investment (2006-2012) of $460.9 million in three projects: Connectivity, Human Development and Productive Development. The Compact’s goal was to advance economic growth and poverty reduction in the Northern Zone of El Salvador. The Productive Development Project of $68 million included three project activities implemented concurrently in the Northern Zone: (i) Production and Business Services ($56 million), (ii) Investment Support ($8 million) and (iii) Financial Services ($4 million). The evaluations summarized here represent the entire Productive Development Project investment, 15% of the total Compact.

Pie Chart: Funding for each component of the Productive Development Project

Program Logic

The Productive Development Project was designed to transition producers to higher-profit activities, generate new investment, expand markets and sales, and create new jobs in ways that would stimulate sustainable economic growth and poverty reduction. The Production and Business Services Activity included on-going technical assistance and training, in-kind donations (starter kits), demonstration plots, and technical and financial support for enterprises created and supported by the project in targeted value chains. 2  The Investment Support Activity was designed to provide investment capital to competitively selected applicants, who, due to insufficient collateral and lack of liquid assets, were not able to finance their investments for business activities located in and benefiting poor inhabitants of the Northern Zone. The Financial Services Activity provided guarantees to support increased lending activity by banks and non-bank financial institutions in the Northern Zone, as well as technical assistance to financial institutions interested in expanding operations and product lines in the Northern Zone. Initially, the Financial Services Activity also included a crop insurance component that was cancelled due to lack of demand.

It was envisioned that the three activities would work together – a portion of Production and Business Services participants would have access to business planning services, investment capital or guaranteed loans through the Investment Support Activity or the Financial Services Activity. The capital and loans would help producers transition to high-value crops and finance new production technologies such as greenhouses and irrigation systems.

Program Logic for Productive Development Project

There were several key assumptions underlying the program logic:

  • Content and duration of training are sufficient to trigger behavior change.
  • Starter kits/in-kind donations are sufficient to trigger sustained behavior change.
  • Producers have necessary access to credit through existing structures supported by the Investment Support Activity or Financial Services Activity.
  • Primary barrier(s) to adoption of improved techniques is lack of knowledge and/or funds for investment.
  • Adoption of improved techniques leads to an increase in productivity.
  • Increases in productivity lead to increases in productive income which, in turn, lead to an increase in overall household income.

It’s important to note that over the course of the Compact, the design of PBS was modified. The first phase (Phase I) of assistance focused on technical assistance with productive activities—particularly milk production in the dairy chain, vegetable production in the horticulture chain, and wood- and clay-based handicraft production in the handicraft chain. After the mid-term review of the Compact, PBS was modified in response to lessons learned during Phase I—namely, that increased and more diversified production was not sufficient to guarantee higher sales and income among participating producers. As such, the second phase (Phase II) of assistance featured more explicit marketing and business development components, including the establishment of two new producer-owned enterprises in the horticulture and dairy chains, and the strengthening of three pre-existing producer-owned enterprises in the handicrafts and dairy chains.

 

Measuring Results

MCC uses multiple sources to measure results, including monitoring data, during Compact implementation, and independent evaluations, which in many cases are continued Post Compact. Monitoring data is typically generated by the implementers, and specifically covers the ‘treatment’ group of farmers and businesses who received training or financial products under the Compact.

The table below includes the monitoring indicators that were tracked during implementation of the project.

Indicators 3

Level

Actual Achieved

Target

Percent Complete

Production and Business Services Activity

Farmers who have applied improved techniques

Outcome

11,520

7,000

165%

Enterprises that have applied improved techniques

Outcome

164

114

144%

Enterprises assisted

Output

272

292

93%

Farmers trained

Output

15,363

10,465

147%

Participants of technical assistance and training – non- agriculture

 

Output

 

2,104

 

3,035

 

69%

Hectares under production with support from the Productive Development Project

 

Output

 

25,399

 

15,000

 

169%

Investment Support Activity

Loan Borrowers

Output

29

N/A

N/A

Loan Borrowers (female)

Output

5

N/A

N/A

Amount of Investment Support fund approved

Output

7,505,299

8,500,000

88.3%

Number of loans executed by the Investment Support Fund

Output

30

N/A

N/A

Number of loans approved by the Investment Support Fund

Output

44

35

125.7%

Financial Services Activity

Number of guarantees granted

Output

5,540

5,109

108.4%

Value of loans guaranteed

Output

12,573,984

9,680,000

129.9%

Value of loans guaranteed – Agriculture

Output

6,230,226

N/A

N/A

Value of loans guaranteed – Non- Agriculture

Output

6,343,759

N/A

N/A

The average completion rate of output and outcome targets is 124% percent; and for 7 of the 10 indicators with targets, those targets were met or exceeded. It should be noted that these numbers are not always the same as the evaluation results because in addition to not taking the “without project scenario” into account as described below, the monitoring data comes from different data sources, data collection instruments, and samples of respondents.

 

Monitoring data is limited in that it cannot tell us what these farmers would have done in the absence of the MCC-funded training, credit, or technical assistance. For example, when implementers report that farmers have exceeded targets around adoption of new techniques, we do not know if these farmers adopted because of the training or would have adopted without the training. This is a key motivation for why MCC invests in independent impact evaluations, which estimate a counterfactual – what would have happened in the absence of the investment. For some activities, impact evaluations are not feasible or cost-effective and in those cases, MCC invests in independent performance evaluations. The evaluations for the Productive Development Project combine the use of impact evaluations and performance evaluations.

Summary of Productive Development Project Evaluations

Component

Evaluation Type

Methodology

Production  and  Business  Services Activity

Impact (Interim) Performance (Final)

Interim Impact: Randomized Roll-out Final Performance: Pre-Post

Production  and  Business  Services Activity – Handicrafts

Impact (Final)

Randomized control trial

Investment Support Activity

Performance (Interim) Performance (Final)

Ex-Post

Financial Services Activity

Performance (Final)

Ex-Post

 

Evaluation Questions

The evaluations of the Productive Development Project were customized for each activity and were designed to answer the following questions:

 

Component

Evaluation Questions

Production   and Business Services Activity

Interim Impact:

  • What was the impact of the Production and Business Services Activity from 2010 to 2011 in the dairy, horticulture and handicrafts value chains on the use of new practices, production, employment creation, and income?

 

Final Performance:

  • How was the PBS Activity designed and why was it designed in this way? What were the key objectives, activities, and outcomes? What was the target population?
  • How was the activity implemented? What were key facilitators and barriers to implementation?
  • Did the activity produce its desired results for production, employment creation, sales and income?
  • Are producer-owned enterprises on a path to sustainability following assistance?

Final Handicrafts Impact:

  • What was the impact of the Production and Business Services Activity from 2009 to 2012 in the handicrafts value chains on the use of new practices, production, employment creation, and income?
  • How do these impacts compare to those detected in the interim evaluation?

Investment Support Activity

Interim Performance:

  • How was the activity designed and why was it designed in this way?
  • How was the activity implemented and did implementation meet initial expectations?
  • Did the activity produce the desired results, including job creation and increased income among loan recipients?

 

Final Performance (combined with Financial Services Activity questions):

  • How were the activities designed? To what extent did the activities’ final designs diverge from the original compact?
  • How were the activities implemented? What were key facilitators and barriers to efficient and effective implementation of FIDENORTE and the guarantee funds?
  • What were the activities’ levels of demand and lending? Did these levels meet original targets? What were the activities’ administrative costs?
  • What were the characteristics of FIDENORTE, PROGARA Norte, and SGR Norte borrowers?
  • What were the characteristics of FIDENORTE loans and loans guaranteed by PROGARA Norte and SGR Norte?
  • What was the overall FIDENORTE repayment rate at key points from 2011 to 2013? What was the default or call-up rate of the guarantee programs?
  • How did FIDENORTE borrowers use credit and technical assistance? What were borrowers’ levels of investment, employment, and income following receipt of FIDENORTE credit?
  • Were there potential effects of the activities on access to credit in the region? On the organizations and FIs that participated in the activities?

Financial Services Activity

Final Performance (combined with Investment Support Activity questions):

  • How were the activities designed? To what extent did the activities’ final designs diverge from the original compact?
  • How were the activities implemented? What were key facilitators and barriers to efficient and effective implementation of FIDENORTE and the guarantee funds?
  • What were the activities’ levels of demand and lending? Did these levels meet original targets? What were the activities’ administrative costs?
  • What were the characteristics of FIDENORTE, PROGARA Norte, and SGR Norte borrowers?
  • What were the characteristics of FIDENORTE loans and loans guaranteed by PROGARA Norte and SGR Norte?
  • What was the overall FIDENORTE repayment rate at key points from 2011 to 2013? What was the default or call-up rate of the guarantee programs?
  • Were there potential effects of the activities on access to credit in the region? On the organizations and FIs that participated in the activities?
  • Are the Financial Services Activity’s guarantee funds sustainable from a financial perspective?

 

Evaluation Results

Productive Development Project Overall

The Productive Development Project activities did not have the level of interaction that was originally expected to occur between them, primarily due to different selection criteria for participants and separate implementers. Administrative records indicate that only 15 PBS participants received loans from the Investment Support Activity out of a total of 30 executed loans, and less than 3 percent of PBS participants received PROGARA Norte-guaranteed loans. Stakeholders generally cited the minimum loan amount of

$50,000 under the Investment Support Activity as a primary reason for the lack of integration between Production and Business Services assistance (which generally served small, poor producers) and the Investment Support Activity (which  generally served  small- and medium-scale business owners). In addition, the PBS Activity had a different implementer than the Investment Support and Financial Services Activities, which made coordination more challenging.

 

Production and Business Services Activity – Interim and Final Impact Evaluation

Although most output and outcome targets for the Production and Business Services Activity were met or exceeded, the independent interim evaluation found varied results for the three value chains. In dairy, the evaluation estimates there were impacts on adoption and increases in farm income. In horticulture, the evaluation estimates impacts on adoption, but no impacts on farm income. In handicrafts, the evaluation estimates impacts on employment for program participants, but no impacts were detected on productive income. In the horticulture evaluation, it should be noted that the sample was underpowered since only about 30 percent of the treatment group enrolled in the training program. This limits the ability to draw conclusions about ultimate impact, though the evaluation still provides ample opportunities for learning.

 

The final data collection rounds for dairy and horticulture were cancelled, however. In handicrafts, additional follow-up data from the final impact evaluation provided more information on the impacts of this activity. The final impact evaluation of the handicrafts sector showed that the increase in employment detected in the 2009-2010 phase of handicrafts implementation (Phase I) had disappeared by the end of Phase II (2012); thus no net impact was detected on employment after nearly three years of assistance. Similarly, no impacts were detected on net handicraft income, net annual household income, or household consumption by the end of Phase II. However, negative impacts were detected on salaried income during Phase II (2011 and 2012). The performance evaluation discussed below provides more information on the results of the handicrafts intervention.

 

The interim evaluation results below capture the phase of training that occurred from 2010 to 2011 for the dairy and horticulture sectors, and 2009-2010 for the handicrafts sector. The final evaluation results capture the results from 2011 to 2012 for the handicrafts sector only.

Evaluator:

Mathematica Policy Research

Methodology

Randomized roll-out

Evaluation Period

12 months for dairy and horticulture; 3 years for handicrafts

Adoption and employment

For the 2010-2011 phase of dairy implementation (Interim):

  • 5 percentage points more likely to conduct quality control
  • 23 percentage points more likely to take measures to reduce costs
  • 7 percentage points more likely to report looking for new clients

For the 2010-2011 phase of horticulture implementation (Interim):

  • 2 percentage points more likely to report selling to enterprises

 

For the 2009-2010 phase of handicrafts implementation (Interim):

  • .13 increase in annual employment generated by program participants (full- time equivalent jobs)

 

For the 2011-2012 phase of handicrafts implementation (Final):

  • The positive effect on employment disappeared by the end of Phase II in 2012

Productive Income

For the 2010-2011 phase of dairy implementation (Interim):

  • $1,849 increase in net annual productive income

 

For the 2010-2011 phase of horticulture implementation (Interim):

  • No impacts detected on net annual productive income

 

For the 2009-2010 phase of handicrafts implementation (Interim):

  • No impacts detected on net annual productive income

 

For the 2011-2012 phase of handicrafts implementation (Final)

  • No impact detected on net annual productive income
  • Negative impacts detected on salaried income consisting of losses of $190 and $177 in 2011 and 2012, respectively

Household Income

For the 2010-2011 phase of dairy implementation (Interim):

  • No impacts detected on net annual household income or consumption by program participants

 

For the 2010-2011 phase of horticulture implementation (Interim):

  • No impacts detected on net annual household income or consumption by program participants

 

For the 2009-2010 phase of handicrafts implementation (Interim):

  • No impacts detected on net annual household income or consumption by program participants

 

For the 2011-2012 phase of handicrafts implementation (Final):

  • Negative impact on net annual household income of $686 in 2011, but no note-worthy impacts on household income were found in 2012
  • No impacts detected on household consumption by program participants

 

Production and Business Services Activity – Final Performance Evaluation

Due to the changes in project design in the middle of implementation and low participation of the treatment group in the horticulture evaluation, MCC cancelled the final data collection rounds for the PBS impact evaluation (for horticulture and dairy; further handicrafts analysis is described above) and decided to conduct a final performance evaluation. The final performance evaluation is unable to provide quantitative estimates of outcomes achieved by the activity; however it provides insights into implementation facilitators and barriers, as well as the potential sustainability of the enterprises supported under the project.

Evaluator:

Mathematica Policy Research

Methodology

Pre-Post

Evaluation Period

2007 to 2012

Implementation Facilitators and Barriers

  • One key facilitator of implementation of the PBS Activity was the large degree of flexibility on the part of implementer staff to modify the PBS assistance model, when stakeholders determined that the activity‘s primary focus on production was not sufficient to achieve desired outcomes.
  • Implementation was constrained by a few factors: large participant targets which resulted in diluted service delivery, lack of intensive assistance related to market access and business development (primarily in Phase I), non-strategic use of donations and Phase II‘s short implementation timeframe.

Production, Employment, Sales

  • Administrative data indicate that PBS assistance surpassed performance targets for increased production, employment, and sales. However, because administrative data do not take into account what would have happened without the project, and the interim impact findings are not generalizable to the full population of PBS participants over the entire PBS implementation period, it is impossible to make a definitive conclusion regarding the impact of the full PBS assistance package from 2008 to 2012.

Sustainability of Enterprises

  • PBS provided training, donations, and organizational and logistical support to all five enterprises in a timely manner. However, the utility of this assistance for dairy and horticulture enterprises is unclear as it appeared to rely on a set of weak assumptions about the enterprises‘ business models. In contrast, efforts to assist pre-existing businesses in the handicrafts value chain appeared largely successful.

Results by Value Chain

  • Horticulture:
    • According to implementers, participants generally experienced increases in production, sales, and income.
    • Some small-scale participants reported deficient technical assistance and suboptimal outcomes.
    • Zamorano was particularly successful in improving production and sales.
  • Dairy:
    • Participants spent less on cattle feed and experienced higher production and sales in the dry season.
    • Among all service providers, TechnoServe and CARE generated particularly strong results.
    • Relatively high-resource participants received more assistance and exhibited better results.
  • Handicrafts:
    • Supported enterprises and artisans reported high satisfaction with assistance and positive results.
    • Workshop owners increased levels of paid labor.

o  Workshop owners benefited substantially from new contracts established under PBS assistance.

 

Investment Support Activity – Interim and Final Performance Evaluations

The Investment Support Activity fell short of its original lending targets; however, interviewed credit recipients experienced higher levels of investment, employment, production and sales than non-credit recipients.  These  results,  however,  are  anecdotal  because  the  evaluation  does  not  have  a  valid comparison group  for loan  recipients or  a  sufficiently  large sample  size  to  attribute  differences in outcomes to the credit.

Evaluator:

Mathematica Policy Research

Methodology

Ex-Post

Evaluation Period

Interim: 2007 to 2011; Final: 2007 to 2013

Implementation

Interim and Final:

  • The Investment Support Activity fell short of its original lending targets— both in value and number of loans approved—due to delays in establishing the trust fund and defining the investment product; a lack of capacity to compose and analyze viable business plans; implementer’s limited experience as a first-tier lender; and a lengthy and complex loan development and approval process.

Characteristics of Borrowers and Loans

Interim and Final:

  • It appears that the Investment Support Activity served its target population of poor producers—particularly poor farmers with some level of working capital—as well as enterprises that benefit poor individuals. In addition, it appears as though the activity complied—at least to some extent—with its original spirit of providing credit to organizations and individuals who otherwise could not have financed their investments.
  • Due to a minimum collateral requirement introduced in 2010, the activity may have missed some key opportunities to serve individuals and organizations with “insufficient collateral and liquid assets to finance their investments” through other means, as mandated in the compact.

Final:

  • Over two-thirds of approved applicants had businesses in the dairy, vegetable farming, or tourism sectors. Across all sectors, the average loan size of approved loans was around $170,000, with an annual interest rate of around 9 percent and a maturity period of 68 months. Loan sizes ranged from $50,000 to $723,000, and interest rates ranged from 8.2 percent to 12.4 percent. 4

Repayment Rate

Final:

  • FIDENORTE loan repayment has steadily worsened since 2012. As of the second quarter of 2014, repayment was approximately 72 percent, down from 78 percent in the second quarter of 2013. The majority of borrowers reported some difficulties in paying back their loans.

Investment,
Employment and Income

 

Interim:

  • Interviewed credit recipients experienced higher levels of investment, employment, production and sales than non-credit recipients. However, the comparisons of recipients and non-recipients merely suggest the potential positive effect of the credit on these key outcomes, as there is no valid comparison group or sufficient sample size to attribute differences to the activity.

  Final

  • Most borrowers reported that FIDENORTE credit played a vital role in stimulating their counterpart investments.
  • Approximately 80 percent of FIDENORTE borrowers (19 of 24 people) said they wouldn’t have been able to generate their current level of employment without FIDENORTE credit.
  • Around 60 percent of FIDENORTE borrowers had profitable businesses three years following receipt of credit, and most borrowers with profitable businesses (67 percent) reported that they could not have achieved this level of success without FIDENORTE. However, around 40 percent of borrowers experienced serious setbacks with their businesses, and faced high FIDENORTE loan payments with tenuous production and revenue streams three years after receiving credit.
  • Given the absence of a valid counterfactual—or an estimate of how borrowers would have fared in the absence of FIDENORTE—we cannot conclude with certainty that FIDENORTE had a positive impact on investment, employment, or income.

Potential Effect on Access to Credit

Final:

  • FIDENORTE appears to have influenced agricultural lending in the country. BANDESAL staff reported that its experience managing FIDENORTE played a large role in its formulation of the Salvadoran Development Bank, a national investment fund introduced in 2012. BANDESAL representatives noted    that the long repayment periods of new credit lines and the inclusion of supplemental working capital in investment loans were direct results of lessons learned from the FIDENORTE trust fund.

 

Financial Services Activity – Final Performance Evaluation

The Financial Services evaluation covers the two guarantee programs, PROGARA Norte and SGR, as well as technical assistance to financial institutions and the failed crop insurance program.

Evaluator:

Mathematica Policy Research

Methodology

Ex-Post

Evaluation Period

2007 to 2015

Implementation

  • PROGARA Norte reached original targets for number of guaranteed loans, but fell short of its goal for value of guaranteed loans. The program’s failure to meet its target value of guaranteed loans, despite meeting its target number of guaranteed loans, reflects overly optimistic initial projections regarding the average value of guaranteed loans.
  • Despite a dearth of potential clients in the Northern Zone, the SGR guarantee provider was able to initiate and formalize a total of 210 guaranteed loans valued at nearly $4.5 million through the SGR Norte fund.

Characteristics of Borrowers and Guaranteed Loans

  • Nearly all PROGARA Norte borrowers were owners of microenterprises with an average of only one employee, including themselves. On average, PROGARA Norte-guaranteed loans were small (average size was around $1,500) and had a maturity period of less than two years. Thirty-four percent of PROGARA Norte-guaranteed loans were provided for basic grain production, largely corn and beans. Less than 10 percent of loans were provided for vegetable production (including cucumber and hot pepper), fruit production (including watermelon and tomato), or dairy production.
  • Microenterprises made up 68 percent of borrowers in the SGR Norte guarantee fund. Loans under the fund averaged around $21,400 and were 3.4 years in length, on average. SGR Norte loans were more likely to be issued for commerce, agriculture, and mining, and less likely to be issued for services, construction, or manufacturing, than the guarantee provider’s non-FOMILENIO loans.

Call-up Rate

  • Call-up rates were relatively low, with loan default and call-up concentrated in one financial institution. Default and call-up of PROGARA Norte guarantees was relatively rare, to the extent that only three percent of guaranteed loans (and one percent of the total value of guaranteed funds) required reimbursement from the guarantee fund.
  • For the SGR Norte fund, default was minimal, with a total of $151,000 in default among $4.5 million in formalized loans (default rate of 3.4 percent).

Potential Effect on Access to Credit

  • Interviewed financial institutions agreed that the guarantee program directly addressed many potential borrowers’ primary constraint of collateral, and that the fund likely reached a demographic of borrowers that likely could not have found credit elsewhere. Financial institutions appear to have strong incentives to continue using guarantee funds like PROGARA Norte.
  • SGR guarantee provider staff asserted that FOMILENIO assistance greatly helped them expand their business in recent years. Although they had not developed a pipeline of new loans in the Northern Zone since the end of the compact period, G&S staff expressed interest in future guarantee programs that would allow them to continue expanding their geographic scope and client profile.

Technical Assistance to Financial Institutions

  • A total of ten financial institutions in the Northern Zone received FOMILENIO-subsidized technical assistance worth approximately $250,000 over the life of the compact, significantly lower than the $1.5 million originally allocated to this activity. Assistance covered a range of projects, including an evaluation of internal controls; an analysis of a new rural microcredit product; improvements to information systems; and training in loan analysis, collection, and customer service.
  • In hindsight, stakeholders noted that the terms of reference—or the parameters of technical assistance that financial institutions could request—may have been too broad, in that they covered a wide range of topics that did not necessarily have a direct connection with the activity’s goal of expanding access to credit.

Crop Insurance

  • Despite its initial budget of $2.6 million, only one farmer took out a FOMILENIO crop insurance policy. A FOMILENIO representative said that cultural issues were the largest obstacle to take-up of insurance policies. An MCC representative added that the conditions of the insurance product were highly complex, and this complexity likely served to deter eligible farmers from pursuing these policies.

 

Lessons Learned

The lessons learned are bundled into two groups below. The first group comes from MCC’s learning around the first five agriculture evaluations released in October 2012. Looking across those five evaluations, and informed by lessons about impact evaluations in agriculture more broadly, MCC identified a set of common lessons. Four of these lessons as illustrated by the El Salvador case are described below. The second group of lessons comes from the performance evaluations for all three activities in the Productive Development Project released later.

  • Always return to the program logic. If the program logic and implementation plan include a variety of value chains, the evaluation must ensure sufficient power to track early and realistic impacts on income in each value chain.  In El Salvador, the evaluation was not originally designed to be done by value chain but by all three sectors together. When unbundled, the design was “underpowered” to report on individual value chains.

 

  • Linking to household income is difficult. In El Salvador dairy, the evaluators find that dairy farmers’ farm incomes roughly double that of the control group; however, they do not find an impact on household income or consumption. This is likely because the number of groups of dairy farmers that were randomized was small, and the evaluation was underpowered to report changes in household income by value chain. This needs to be taken into consideration for future evaluation design.

 

  • Test traditional assumptions. In El Salvador, some of the evaluation findings suggest that tailored trainings and donations may produce better results in the short-term. However, the project and evaluation were not designed to test effects of variation in training content or duration in order to confirm this. MCC and MCAs will look for future opportunities to use impact evaluations to test assumptions around the appropriate content and duration of training to maximize impact.

 

  • The randomized roll-out evaluation approach has risks. In a randomized roll-out approach, a first round of treatment farmers is compared to a control group of farmers that receive training at a later date. The key to this approach is that there be enough time between the two phases to see behavior change and accrual of benefits for the first farmers before the second round of farmers is trained. In the case of the handicrafts project, more was learned with the follow-up data and impact analysis   on intermediate and final outcomes. For the other value chains, however, the control groups have been trained as per the agreed roll-out methodology and additional learning using these evaluations is limited.

 

The following additional lessons have been identified through MCC’s review of the performance evaluations for all three activities of the Productive Development Project.

  • When important for unbundling program results, require the reporting of detailed cost information. Over $10 million was available for donations to beneficiaries under the Production and Business Services Activity. However, MCC did not require MCA and its implementer to report in  detail the amount of donations that were provided to individual farmers or enterprises. Detailed records were kept by the implementer; however, only high-level aggregated numbers were reported back to MCC. This has resulted in the inability of the evaluation to analyze who benefited the most from donations and whether or not receiving large amount of donations was correlated with improved outcomes. To the extent that MCC wants to analyze this type of information in future projects, detailed reporting on costs from implementers should be required by their contracts and potentially required from accountable entities as well.
  • The activity’s objective, target population, type of intervention, definitions, selection methodology, and expected results should be defined prior to investment. While these were stated in some form in the Compact for the Investment Support Activity, the definitions were not clear up-front or shared by all of the stakeholders. As a result, the interpretation of this language was debated throughout implementation, affecting the size range of the investments, the interest rate and the collateral requirements. In future circumstances, MCC should be very clear when drafting investment related language in Compacts in order to set out the purpose, activities and expected results of the intervention, which should be accompanied by a detailed term sheet to guide implementation preparation and investment.
  • Starting up a new financial product or service takes time and should be done as early as possible in the Compact or existing services with a proven track record should be expanded. There is a relatively long lead time involved in creating the documents, institutional arrangements, policies,  and approval processes that are prerequisites to making lending and other financial service products operational and available. These also precede the process of letting potential beneficiaries know of the availability of such products and services. If possible, these mechanisms and arrangements should be completed prior to or at the very beginning of the activation of a Compact. In El Salvador they were not and that contributed to a lower level of lending activity through FIDENORTE. The partial loan guarantees offered through PROGARA Norte, in contrast, met the target for the number of guarantees and attributed that largely to its pre-established processes and organizational structure. In future compacts, MCC should similarly seek to expand existing financial networks that have a demonstrated track record.
     
  • Implementer capacity matters. For the Investment Support Activity, the relationship between BMI and FOMILENIO was governed through the trust agreement and an Implementing Entity Agreement (IEA) but compliance and enforcement of the agreements was a struggle throughout implementation.  In hindsight, there should have been even better management of the IEA, more performance based incentives, and potentially some technical assistance to ensure that MCC funds were used most productively. This was a missed opportunity for BMI, FOMILENIO, and MCC to invest in more SMEs in the Northern Zone.
     
  • Overhead costs need to be considered when designing the size of financial activities. At the close of the program, the Investment Support Activity ERR was lower than anticipated. This was partially because it cost the same amount of money to approve $7.5 million in loans as it would have cost to execute the larger planned program. Overhead costs need to be a consideration in sizing such activities.
     
  • Linkages between activities will not happen on their own. The design of the Productive Development Project included three activities. In particular, it was envisioned that the Production and Business Services Activity (PBS) and the Investment Support Activity would work together. Producers receiving technical assistance and training under PBS were to receive help developing business plans to access credit under the Investment Support Activity. Administrative records indicate that only 15 PBS participants received loans from the Investment Support Activity out of a total of 30 executed loans. This was not the level of interaction originally envisioned between the two activities. Incentives or requirements could have been included in implementer contracts to ensure that the two activities worked together. In addition, the targeted beneficiaries of each activity could have been aligned so that there was more overlap. The minimum loan amount of $50,000 under the Investment Support Activity may have been the primary reason for the lack of integration between PBS (which generally served small, poor producers) and the Investment Support Activity (which generally served small- and medium-scale business owners).

Next Steps

  • The evaluation reports, in conjunction with anonymized data sets and associated supporting technical documentation, will be available on the MCC external web site for public access and use in 2016; refer to the MCC Evaluation Catalog available at: http://data.mcc.gov/evaluations/index.php/catalog.
Footnotes
  • 1. This Summary of Findings has been updated to include final impact evaluation results for the handicrafts value chain and subsequently updated to include final performance evaluation results for the Investment Support and Financial Services Activities.
  • 2. The evaluations for the Production and Business Services Activity cover only the dairy, horticulture and handicrafts. These are three of the eight value chains targeted in the investment, and the majority of the investment.
  • 3. Note that the baseline is 0 for all of these indicators.
  • 4. The interest rate on loans varied with the maturity of the loan and whether the borrower was granted a grace period on repayments. The loans were intended to be made at unsubsidized interest rates.