Measuring Results of the Mozambique Farmer Income Support Project

In Context

The MCC compact with Mozambique was a five-year investment (2008-2013) of $447.1 million designed to increase economic growth and reduce poverty in the Northern provinces of Cabo Delgado, Nampula, Zambézia, and Niassa. The compact funded four major projects: 1) Water Supply and Sanitation ($200.2m), 2) Rehabilitation/Construction of Roads ($136.8m), 3) Land Tenure Services ($39.5m), and 4) Farmer Income Support ($18.9m). The Farmer Income Support Project (FISP) is the subject of both the results described here and an independent performance evaluation, to be released by MCC in 2016.

The objective of the FISP project was to address the adverse impacts of Coconut Lethal Yellowing Disease (CLYD), which threatens an important source of income for over 1.7 million people living in Zambézia and Nampula provinces. The Farmer Income Support Project was comprised of four key activities: 1) Rehabilitation of Endemic Areas; 2) Control of Epidemic Disease; 3) Business Development Support; and 4) Research and Development Support. This component represents 4 percent of the total compact..

                        

*These figures are based on MCC disbursements (Compact and Compact Implementation Funds as of September 2014.)

Program Logic

FISP focused on mitigation of CLYD which, based on the rate of disease spread in 2007, was projected to impact over 50 percent of coconut trees in Zambézia and Nampula by 2017. FISP targeted smallholder farmers, many of whom had been operating as outgrowers for several, privately-operated coconut plantations in the area. FISP’s approach for mitigating the disease and its potential negative impact on the current and future incomes of the populations in these provinces was to identify geographic zones based on disease prevalence (epidemic or endemic) and implement a different set of activities in each zone.

 

 

FISP Intervention Zones

<10% disease prevalence

>75% disease prevalence

 

Project Activities

Control of Epidemic Disease ($6.9m)

Rehabilitation of Endemic Areas ($8.9m)

  • Removal (cut and burn) of diseased trees to control disease spread
  • Replacement of lost trees with CLYD-tolerant seedlings
  • Large-scale clearing of diseased and dead trees
  • Facilitate farmer adoption of alternative cropping systems (technical assistance and provision of improved seeds)
  • Replacement of lost trees with CLYD-tolerant seedlings

Business Development Fund ($0.8m)

Small business grants to support entrepreneurship for income diversification and development of the coconut industry value chain, based on coconut wood products

Research and Development ($2.5m)

Support to establish scientific protocols for developing and testing CLYD-tolerant varieties of coconut

 

Under the FISP program logic, epidemic zones were still considered to be a productive coconut-growing area, so the main focus of the activities in this zone was to mitigate disease spread. The endemic zone activity, on the other hand, focused on income diversification to create a way for smallholders to generate agricultural income in the short run (while replacement coconut seedlings grew to productive maturity).

                   

 

There were several key assumptions underlying the FISP program logic:

  • In endemic zones, it was assumed that smallholders would have access to viable outlets for the sale of the alternative crops they were trained and supplied to cultivate.
  • It was assumed that cutting and burning diseased trees along with training on correct disease identification in targeted epidemic zone communities would lessen the prevalence of CLYD and support coconut productivity across the entire region. Dead trees, which are associated with rhinoceros beetle infestation that can also kill coconut trees, were not removed in epidemic areas.
  • Although there were no conclusive scientific studies confirming that the variety of coconut plant distributed by the program was CLYD-tolerant over the long-term, it was assumed that the coconut seedlings planted in both endemic and epidemic zones would provide a future stock of trees that would be less vulnerable to CLYD.
  • FISP activities were exclusively smallholder-focused, since neither the Mozambican government nor the farmers themselves had the necessary resources available to control CLYD. FISP did not directly target private coconut plantations because they were assumed to have had the resources available to pursue abatement independently.
  • Because FISP was not scaled to work in all areas where CLYD was present, it was designed to be implemented alongside parallel efforts to combat CLYD on large coconut plantations. It was assumed that that FISP activities would incentivize on-planation control by providing assurance that privately-funded disease abatement efforts not be undermined by unchecked disease in neighboring smallholder areas. Although FISP had focused on securing their co-operation during project start-up and implementation, plantation owners did not ultimately engage in any concerted cutting or burning effort. Instead, many abandoned coconut production entirely, shifting instead to other crops and industries.
  • The Business Development Fund (BDF) assumed that small grants would act as catalysts to spur innovation and create markets for coconut by-products, especially wood and other parts of the coconut that are not typically used.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. This limitation of monitoring data is a key reason why MCC invests in impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence to assess the likely impact of MCC investments on key outcomes.

 

Monitoring Results

The following table summarizes performance on key output and outcome indicators specific to FISP.

Indicators

Indicator Level

Baseline (2008)

Actual Achieved (2013)

Target

Percent Complete

Coconut seedlings planted

Output

0

782,609

650,000

120

Survival rate of Coconut seedlings (%)

Outcome

0

76

80

95

Hectares with dead trees cleared in endemic zones

Output

0

8,000

8,000

100

Diseased/dead palms trees cut in epidemic zones

Output

0

600,000

600,000

100

Farmers trained in surveillance and pest/ disease control for coconuts

 

Output

 

0

 

15,607

 

8,000

 

195

Farmers adopting techniques in surveillance and pest/disease control for coconuts (%)

 

Outcome

 

0

 

44

 

30

 

147

Farmers trained in planting and post-planting management of coconuts

 

Output

 

0

 

28,830

 

8,000

 

360

Farmers adopting planting and post-planting management techniques of coconuts (%)

 

Outcome

 

0

 

32

 

60

 

53

Farmers trained in alternative crop production

Output

0

8,958

8,000

112

Farmers using alternative crops techniques (%)

Outcome

0

38

30

127

Hectares of alternative crops under production

Output

0

7,686

8,000

96

Businesses receiving BDF grants

Output

0

119

150

79

Source: (December 2013 ITT, based on reporting from the FISP Service Provider). Please refer to the Mozambique Closeout ITT for aggregated and disaggregated data.

 

The average completion rate of output and outcome targets is 132 percent; and in 8 of the 12 indicators with targets, targets were met or exceeded.

Evaluation Questions

The evaluation was designed to answer questions such as:

  1. What is the impact of FISP on reducing  CLYD prevalence and spread in project areas?
  2. What is the impact of FISP on coconut production?
  3. What is the impact of FISP on the cultivation of alternative crops?
  4. What is the impact of FISP on the present survival rate of the seedlings?
  5. What is the impact of FISP on the incomes of participating farmers in the endemic and epidemic zones?
  6. What is FISP's current economic rate of return?

Evaluation Results

The overall evaluation used a mixed-methods approach to evaluate the overall performance of FISP. Two different quantitative evaluation designs were used to measure the impact of FISP on tree health and household income in the epidemic and endemic zones. In both areas the impact of FISP was estimated by comparing outcomes in the project implementation areas with outcomes in geographic areas outside of the project areas. Case studies were used to assess the BDF grants and the cross-cutting research and development (R&D) activity. The evaluation of all FISP components—the three activities that supported the epidemic zone and endemic zone interventions, the BDF activity, and the R&D activity—finds that the project was partially successful, although sustainability issues could stand to dampen these success over the long run.

 

Epidemic Zone Interventions

In the epidemic zone FISP was successful in reducing overall disease prevalence. Despite the success in efforts to reduce disease prevalence, coconut production did not increase enough to lead to increased household income from the sale of coconuts. Instead, FISP had an impact on non-farm income (fishing and non-skilled labor). The evaluator hypothesizes that the mechanism for this unanticipated outcome is that by alerting farmers to the lethality of CLYD, FISP may have induced coconut farmers to diversify sooner to non-agricultural sources of income.

Evaluator

Abt Associates

Methodology

A quasi-experimental “border discontinuity” design where a FISP- designated phytosanitary barrier established project intervention and comparison areas

Evaluation Period

2014 (12-24 months after the FISP activities ended)

Impact on CLYD Prevalence and Mitigation

  • The proportion of dead trees/trees with CLYD in project areas (32 percent) was 18 points lower than comparison areas (49 percent)
  • Project area farmers planted over two times as many coconut tree seedlings as comparison farmers (4 vs. 1.5)
    • No impact on seedling survival rate
  • No impact on knowledge of proper CLYD mitigation methods

Impact on Coconut Production

  • Project area households produced 90 kgs more of coconuts annually than comparison households
    • Overall production in project areas in 2014 (123 kgs/year) was lower than reported production before the program start in 2009 (401 kgs/year)
    • Total coconut production was significantly lower for female- headed households

Impact on Household Income

  • No significant impact on income from coconut sales
  • Significant impact non-farm income (US$85.30/year)

 

Endemic Zone Interventions

FISP had a measurable and significant impact on households’ adoption of alternative crops, which led to increased production of FISP-promoted crops. However, the magnitude of the increase was small and did not result in an impact on household income

Evaluator

Abt Associates

Methodology

A non-experimental matching method for selecting geographic comparison areas that were very similar to the treated project areas, in terms of baseline disease prevalence and distance from the coast.

Evaluation Period

2014 (12-24 months after the FISP activities ended)

Impact on CLYD Mitigation

  • Project area farmers planted over three times as many coconut tree seedlings as comparison farmers (8.5 vs. 2.3)
    • No impact on seedling survival rate
  • Project area farmers were significantly more likely to know that cutting trees is the best way to mitigate CLYD

Impact on Alternative Crop Adoption

  • Project area farmers were 70 percent more likely to plant one of the FISP alternative crops than comparison households

Impact on Alternative Crop Revenue

  • Though statistically significant, revenue from alternative crops was negligible (less than 0.2% of average annual income)

Household Income

  • No significant impact on- or off-farm income

 

Cross-Cutting Activities

The evaluation of the R&D activity suggests that a more focused strategy could have yielded better results. The R&D activity had a diverse set of objectives, making it hard for FISP to focus on any one of them for strong results. Overall the activity fostered no peer-reviewed scientific output and although establishment of molecular diagnostic capacity for CLYD was achieved, it was not sustained after the end of the program. The BDF grant program was perceived by grant recipients to be beneficial, however, no beneficiaries were able to provide information to quantify increases in sales, net income, or employment.

 

Current Economic Rate of Return

The evaluator’s calculation of the project’s current ERR was based on the quantitative impact estimates of disease prevalence and seedling survival rate with and without the project in epidemic areas, and alternative crop uptake and seedling survival rate with and without the project in endemic areas. Sensitivity analysis was also conducted to explore outcomes in the event that CLYD resurfaces in the surviving coconut seedlings, given that disease resistance of the coconut cultivar used for replanting was not scientifically confirmed. This revised model produced a more modest overall ERR of 16.4 percent, compared to MCC’s end-of-project ERR estimate of 36 percent. The reduced ERR is driven by the higher observed disease prevalence rates and lower seedling survival rates than those anticipated by MCC at project closeout.

 

Conclusions and Implications

  • Although cutting trees and burning tree stumps reduced CLYD prevalence in epidemic areas, it was not enough to restore production yields to historic levels. While FISP’s strategy of tree cutting based on visual identification of CLYD was effective in controlling the disease, two years after the activity ended, the disease control was not at a level that could sustain households on coconut-derived incomes alone.
  • In the absence of cost-effective approaches to cutting trees that stakeholders could realistically sustain beyond the end of the program, a reoccurrence of CLYD in the future is likely. Although local stakeholders did not have the financial resources available to sustain FISP as it was implemented, community engagement could be a useful for sustaining the gains made to control CLYD. FISP itself moved to engaging communities in the later years, and determined that engaging communities could be a more-sustainable and cost-effective approach to CLYD abatement, although community safety in tree removal would need to be addressed.
  • FISP’s decision not to cut down dead trees or address large swaths of dead trees on private plantations in the epidemic zone may have contributed to the low survival-rate of seedlings. Rhinoceros beetles, which thrive in dead, CLYD-affected trees, weaken and kill coconut trees, particularly immature trees. This suggests that achievement of reductions in disease prevalence requires an approach that directly addresses all sources of disease and infestation; successful disease abatement in resource-limited contexts may require implementers to limiting the area of operation.
  • Coconut replanting efforts may be more effective in the long run after complete control of both CLYD and beetle infestation and with better evidence on CLYD tolerance of new, replanted varieties. As measured two years after the program ended, FISP did not have a significant impact on seedling survival, and the survival rate was lower than what FISP intended. Seedling replanting efforts may be more appropriate after both CLYD and beetle infestation are adequately addressed; otherwise, replanting efforts may be affected by loss of seedlings to beetle infestation. Furthermore, it is important to ensure that the seedlings are tolerant of CLYD with established scientific evidence; otherwise, replanting efforts may not yield anticipated returns.
  • Efforts to diversify households’ income sources require a multi-sectoral approach. In areas where populations have traditionally been dependent on coconut production, households are not used to labor-intensive alternative crops. As a result, efforts to convince households to diversify crops may result in modest income increases. Disease control efforts may only bring temporary relief and not to a level to allow reliance on coconut production for incomes. Therefore, future programs to mitigate the income impact of CLYD infestation in coconut-growing areas should create opportunities for the households in non-farming sectors.
  • Gender constraints are important. Seedling care, planting, and care of alternative crops are often activities that women have to add on to their already burdened rural lives. The design of programs that introduce such activities must consider the constraints faced by women and consider their overall scarcity of time. Under FISP, these constraints affected the success of seedling replanting efforts.

 

MCC Lessons Learned

  • Balance project complexity with MCC’s capacity and strengths. By design, FISP was a complex project with a programmatic focus that was somewhat misaligned with MCC’s core expertise and experience at the time the program was launched. As a result, project stakeholders perceive that FISP required a disproportionately-intensive effort from MCC and its partners to ensure successful implementation. Considering the mixed evaluation results and modest ERR, however, this extra effort may have been better expended on other aspects of the Compact. Experiences like FISP have motivated MCC to define and develop its core competencies, and take on “lighter touch” interventions on efforts outside these areas.

 

  • Firm-fixed-price contracts can complicate program implementation. Implementation of the various FISP components was achieved through firm-fixed-price (FFP) contracts with private service providers. In Mozambique, as well as other early MCC Compacts, FFP contracts were a commonly-used mechanism to procure project implementation services. Although FFPs can be effective tools to support budget and cost control, their disadvantage is that they do not provide donors with much control and/or flexibility over how the funds are used. Several FISP stakeholders have noted that FFP contracts were a hindrance to implementation given that that MCC and its partners were continuously grappling with various dynamic contextual and epidemiological factors affecting the program logic and design. In part because of the experiences like FISP, MCC now advises that a wider variety of contracts is considered to procure project implementation services.

 

  • Consideration of market forces is critical. FISP intended to help smallholders preserve historic sources of household income and promote income diversification through the sale of alternative crops. It did not, however, directly address the broader market forces that could support or detract from the program aims, which may have detracted from the program’s impact. For example, in the epidemic zone, a focus on keeping farmers in coconut production may have been a short-sighted goal given the wider trends in the Mozambican coconut industry, namely the shuttering of private plantations in response to disease and economic pressures, and a changing international coconut product market that favored produce from other countries. In the endemic zones, a lack of formal linkages to alternative crop markets may have contributed to the low-intensity in alternative crop-uptake among farmers. Today, economic constraints and root cause analyses, which are intended to identify and consider a broad set of economic factors impacting potential intervention areas, are a regular part of Compact Development at MCC and contribute to a more comprehensive understanding contextual factors which could impact a project’s logic model. Lessons from the FISP experience support the continued use of carefully and discretely applied analytical tools in future MCC compacts.

 

  • Harness opportunities to support on- and off-farm income generation. In the right conditions with the right inputs smallholder farmers can realize on-farm income gains through improved productivity, enhanced market linkages, or sales of higher-value produce. In the case of FISP, observed off-farm income gains in epidemic zones and weak alternative crop uptake in endemic zones suggests the need for responsive interventions that are sensitive to scenarios in which on-farm gains fail to materialize or are hard to sustain. Going forward project teams may want to explore whether traditional strategies to promote improved on-farm productivity among smallholder and subsistence farmers might also be bolstered by solutions that develop, support, and sustain off-farm income generation potential among targeted beneficiaries.

 

  • Plan for contingencies in case key program assumptions fail. Although MCC attempted to foster buy- in from plantation owners to bolster FISP’s viability, the long-term sustainability of FISP benefits were negatively affected by the plantation owners’ unanticipated divergence away from the coconut industry. Apart from an increased risk of CLYD re-occurrence and beetle infestation, this development weakened smallholder outgrowers’ market linkages and sent MCC scrambling to try to find another partner that could fill the void left by the plantations. Recognizing that the Government of Mozambique was not positioned to sustain FISP gains in the post-Compact period, MCC tried to generate interest among other private sector players to invest in the continuation of the disease control activities. Despite these concerted efforts, MCC ultimately failed to identify a willing partner before the close of the Compact. The FISP experience provides a cautionary tale about anchoring a program’s long-term sustainability to the ongoing cooperation of the private sector. Considering that MCC has less leverage over these actors, early integration of contingency planning in the event that key assumptions undergirding the program logic fail can promote and secure long-term investment sustainability. Many of MCC’s subsequent Compact activities incorporate strategies that support such an approach including sustainability planning, capacity building, and coordination.