Measuring Results of the Morocco Rain-Fed Olive Tree Rehabilitation Sub-Activity

A Sub Activity of the Fruit Tree Productivity Project

The $129.7 million Rain-Fed Olive, Almond & Fig Tree Activity represented 39% of the Fruit Tree Productivity Project and 20% of the total compact. The Rehabilitation and Intensification of Rain-fed Olive Trees Sub-activity was one of two components of this activity, which provided technical support and guidance to farmers (olive producers), professional agricultural organizations (OPA), and processing units (UT) in the value chain of the olive industry. This sub-activity is the subject of an ongoing independent performance and impact evaluation, the intermediate results of which are summarized below. In terms of the OPAs and processing units, positive developments had been observed along the value chain of the Moroccan olive sector (although any observed change cannot be clearly attributed to the intervention since an impact evaluation was not carried out for these two groups). As for farmers, while training targets were generally achieved with an 89% achievement rate, those who participated in the training did not universally adopt good practices. The impact evaluation used “differences in differences” to look deeper into the impacts of the intervention on the farmers and detected no improvements in production, productivity, or agricultural revenue by the end of the project, but did find a positive impact on the adoption of some of the best practices that were part of training, such as pruning, water management practices, and use of specialized equipment for harvesting.  However, with the exception of harvesting equipment usage, analysis was not able to determine conclusively that the adoption of these best practices could be attributed to the program. The key limitation of this evaluation was that the assessment covers only one or, at most, two growing seasons since the training. Farmers may continue to increasingly adopt best practices over time, and certain practices such as pruning may not begin to increase yields for several growing seasons. Continued evaluation of the impacts of this sub-activity is underway and a follow-up evaluation report is expected in 2017.

In Context

Pie Chart: Distribution of Funding of Morocco I Compact

The MCC compact with Morocco was a five-year investment (2008-2013) of $649.4 million in five projects:  Fruit Tree Productivity (PAF, per its acronym in French), Small Scale Fisheries, Artisan and Fez Medina, Financial Services, and Enterprise Support.  

The $323 million PAF Project represented 49% of the total compact and operated through five activities: (i) Rain-fed Olive, Almond, and Fig Tree Intensification and Expansion, (ii) Olive Tree Irrigation and Intensification in PMH zones, (iii) Date Tree Irrigation and Intensification in the Oasis, (iv) Fruit Tree Sector Services, (v) the creation of new modern crushing units supported by a dedicated fund (Catalyst Fund).

The Rehabilitation and Intensification of Rain-fed Olive Trees Sub-activity provided technical support and guidance to the farmers, professional organizations, and other agents in the value chain of the olive industry. These services were structured into three areas of intervention:

  • Pre-production phase (upstream): Training, technical support, and guidance for farmers, youth farmers, and professional organizations
  • Post-production phase (downstream): Targeted support for training and technical support to stakeholders involved in the processing, valuation, and marketing of fruit tree products
  • Pre-production and post-production sectors: Support the identification, development, and implementation of pilot projects to benefit women and women’s organizations.

 

This sub-activity is the subject of an ongoing performance and independent impact evaluation, the intermediate results of which are summarized below.

Program Logic

The Fruit Tree Productivity Project (PAF) was conceived to stimulate growth in the agriculture sector through the creation of the necessary conditions for increasing the productivity and competitiveness of the trees sectors, and therefore contributing to improved economic growth and a reduction in poverty in the concerned provinces.

The rehabilitation sub-activities (intervention) were targeted at three major populations: farmers (olive producers), professional agricultural organizations (OPA), and processing units (UT). The intervention consisted mainly of technical assistance activities including training and guidance/coaching for the three groups. The farmers were trained in the best techniques for production and harvesting with the goal of increasing yields and improving the quality and market value of their product. In order to have a more effective implementation, the intervention also sought to grow and strengthen the capacity of the OPAs so that they could better organize the farmers to improve farmers’ bargaining power and sales. Finally, in order to achieve a better quality oil, the intervention planned to oversee the upgrading of a number of crushing units, to ensure that they obeyed food safety and environmental laws.

 

The program logic of the activity is described in the following figure:

 

The following were some of the critical assumptions underlying the program logic:

  • Capable technical assistance will be provided
  • Farmers/firms have the incentive to participate (ie. they see the value in the training)
  • There will be demonstration effects (knowledge diffusion) from early adopters to non-adopters
  • Both Upstream (production) improvements and downstream (commercialization) improvements happen simultaneously (i.e. commercialization improvements create demand for high-quality olives and supply of high quality olives allows for improve olive oil production)

Measuring Results

MCC uses multiple sources to measure results. Monitoring data is used during compact implementation.  Independent evaluations are generally completed post-compact. Monitoring data is typically generated by project implementers, and specifically covers the project participants who received treatment through the compact. However, monitoring data is limited in that it cannot tell us what the project participants would have done in the absence of the MCC-funded training. This is a key motivation why MCC invests in independent impact evaluations. This piece of information, known as a counterfactual, enables an evaluator to determine whether the intervention has had a significant impact on beneficiaries and achieved the expected outcomes.

 

Monitoring Results

The following table summarizes the performance of specific monitoring indicators for this sub-activity.

 

Indicators

Level

Actual

Target

Percent Achieved

Number of farmers trained

Output

40,863

42,000

97%

Number of youth farmers trained

Output

15,476

7,500

206%

Number of OPA members trained

Output

4,172

2,880

145%

Number of processing units assisted

Output

114

114

100%

Number of pilot projects for women’s organizations receiving support

Output

14

14

100%

Number of processing units that have applied good manufacturing practices and hygiene

Outcome

34

114

30%

 

The average completion rate for these indicators is 113%, and the target values were met or exceeded for 4 out of 6 indicators.

 

Evaluation Questions

The evaluation for the rain-fed olive rehabilitation sub-activity included both performance evaluation questions as well as rigorous impact evaluation questions.

 

The performance evaluation focused on the activities of training, technical support, and mentoring of farmers, their professional agricultural organizations, and the processing units, including the Catalyst Fund component that was added to the program in 2011 in the rain-fed area. The principal research questions for the performance evaluation were:

  • What is the validity of the intervention logic, and what are its underlying hypotheses?
  • What are the success factors and the impeding factors with respect to the achievement of the program objectives?
  • Did the project achieve the objectives initially defined in the logical framework?
  • What are the strengths and weaknesses of the implementation of the program?
  • What are the opportunities and threats that may work for or against the sustainability of the results achieved?

 

The rigorous impact evaluation of the olive tree orchard rehabilitation in rain-fed areas is based on measurement of the net change in the agricultural income of olive-producer households that can be attributed to the intervention. The principal research questions for the impact evaluation were:

  • What is the impact of the rehabilitation of the olive trees component on farmers’ revenues in the targeted rain-fed zones?
  • Has the production of olives increased in volume or value?
  • Were the promoted techniques adopted by the farmers?
  • Has there been an improvement in the quality of olive oil produced?
  • Have the professional agricultural organizations played their role in the development of the olive oil sector?
  • Has the knowledge generated by the project diffused beyond the project’s “treatment” perimeters?
  • What are the differences in the impact of the project by sex, age, and income level?
  • What are the unintended impacts of rehabilitation?

The major principle of an impact evaluation is the construction of a counterfactual, namely to identify what would have happened if the intervention had not taken place. To create this counterfactual, NORC collected data from matched perimeters 1  that were randomly assigned to either the treatment group, which received the intervention, or the control group, which did not receive the intervention. The evaluation is designed to capture the differences between the treatment and control perimeters, as well as differences over time, from the baseline in 2010 to the end of monitoring in 2013 2 . The evaluation plan used the “differences in differences” or “double differences” method, using data obtained through surveys of farmers in the treatment and control perimeters conducted in 2011, 2012, and 2013. This data was collected in approximately 140 treatment and control perimeters, with about 2,500 farmers.

 

Evaluation Results

The impact and performance evaluations for this project indicate that results were mixed at the end of the MCA-Morocco Compact.

The performance evaluation components of the report found that while training targets were generally achieved with an 89% achievement rate, through the training of 29,481 individual farmers in at least one of the four modules in 248 perimeter, those who participated in the training did not universally adopt good practices. One reason for this was that the trainings were not delivered in a seasonally time-appropriate manner. One respondent to the performance evaluation surveys claimed that the trainings would have been better received if they had been given during the time of the harvest, the point where the potential for most of the farmers’ revenue exists. The training intervention had a delay of three months, so the timing of the delivery of training interfered with other activities. In focus group discussions, several beneficiaries, most notably the young, claimed that both the number of sessions and length of trainings were not adequate to meet the goals of the project. For example, sessions on agriculture only lasted for half of a day, which is not adequate for learning the necessary techniques. Young people also expressed an interest in receiving more than just information sessions, and were looking for an experience that would approximate an apprenticeship or practical job supervision. While beneficiaries were overall pleased with the training, several performance improvements could be made in the future.

 

In terms of the OPAs and processing units, positive developments had been observed along the value chain of the Moroccan olive sector (although any observed change cannot be clearly attributed to the intervention since an impact evaluation was not carried out for these two groups).  Some OPAs that participated in training had improved some of their services to their members. Likewise, some managers of the processing units who participated in training had improved their knowledge about the production of olive oil. Indeed, 34 of the 110 processing units, who developed a business plan with the support of technical assistance, had signed commitment statements to improve their crushing units. Although the intervention was more focused on the GIEs and on assisting in the creation of new processing units rather than on upgrading previously existing private processing units, the improvement in knowledge of best practices for olive oil production by private processing units was a positive result of the intervention.

Performance Evaluation

Evaluator

National Opinion Research Center

Period of Evaluation

From December 2009 to August 2013

Intermediate Effects

  • 29,481 individual farmers participated in at least one of four training modules in 248 perimeters, an 89% achievement rate
  • However, those who participated did not universally adopt best practices because trainings were not as effective as they could have been
  • Positive developments were observed along the Olive value chain in terms of the OPAs and processing units
  • 34 of 110 processing units who had developed a business plan with the support of technical assistance signed commitment statements to improve crushing units

Final Impact

No impact on farmers’ revenue

 

 

Looking at the impact evaluation, no improvements in production, productivity, or agricultural revenue were detected by the end of the project. The analysis found a positive impact on the adoption of some of the best practices that were part of training, such as pruning, water management practices, and use of specialized equipment for harvesting.  However, with the exception of harvesting equipment usage, analysis was not able to determine conclusively that the adoption of these best practices was caused the program, as opposed to practices the farmers would have adopted anyway. One reason for the lack of adoption of some best practices was the lack of financial assistance or provision of material necessary for adoption. Farmers did not have the means to procure the tools and equipment needed to put into practice the entire set of practical measures. The short timeframe of the analysis is like an important contributing factor to lack of impact found by the analysis.  The project training took place in 2011 and 2012, and the last survey was conducted at the beginning of 2013. Thus, the assessment covers only one or, at most, two growing seasons since the training. Farmers may continue to increasingly adopt best practices over time, and in addition certain practices such as pruning do not begin to increase yields for several growing seasons. A future evaluation after one or two olive sector cycles could more definitively detect an increase in the adoption rate of best practices, and an impact on production and possibly agricultural revenue.

 

Impact Evaluation

Evaluator

National Opinion Research Center

Methodology   

Randomized controlled trial approach (RCT) (“matched-pairs cluster randomization design”) with application of difference-in-differences method.

Period of Evaluation

From December 2009 to August 2013

Intermediate Effects

  • Increased adoption of several good practices techniques explained during training
    • Using manure: increased 23.4%
    • Pruning: increased 38.8%
    • Harvesting with vibrators: increased 191.9%
    • Digging impluviums: increased 166%
  • No improvement in the adoption rate of other best practices including:
    • Manufacture of impluviums
    • Use of tarpaulins during harvest
    • Storage of harvested olives in boxes
    • Use of fertilizers and pesticides
    • Storage of oil in plastic food-grade containers
    • Use of modern crushing units
    • No improvement in olive yields

Final Impact

No impact on farmers’ revenues

 

 

 

Consequently, the Rehabilitation and Intensification of the Rain-Fed Olive Trees Sub-activity was a key first step in the development of the sector, through training, mentoring, and organization of farmers into cooperatives and EIGS capable of producing virgin and extra virgin olive oil, but continued assistance and investment in the olive value chain will be necessary in order to achieve full effectiveness of the project. The project’s hypothesis is that farmers will adopt these best practices gradually with market integration and startup of the Catalyst Fund processing units.

Lessons Learned

The performance and impact evaluations highlighted some lessons learned that should be taken into consideration during the conception of future similar projects. In particular:

 

  • Practical trainings are more effective than theoretical trainings. Because of the reality of the Moroccan market and the demand of previous beneficiaries, the pedagogical approach should include more practical training sessions rather than theoretical training sessions in the classroom.

 

  • Solid management and coordination is needed. The program lacked an adequate number of managers to oversee the trainings. This stems from the lack of funding, and the lack of an assured contract beyond a period of six months. The amount of travel required by professional organization managers and the obvious lack of job security (noted by the lack of contract security and Morocco’s high propensity to unemployment) transferred into very few incentives for people to perform this job function. Additionally, the management and coordination of the program implementers were very weak. The employees who were placed in the field complained of little feedback from those located at headquarters. Key informant interviews revealed that poor communication between UNOPS and Morocco’s Center of Labor and the Provincial Director of Agriculture created tensions at the management level. While improved budget allocation may improve some of the financial incentives for managers, better management of the program’s implementers would pose less risk to the project’s implementation.

 

  • Easing the burden for beneficiaries provides larger incentives. The introduction of the Catalyst Fund near the program’s mid-point was a significant methodological improvement and addition to the project. By enabling the professional agricultural organizations to have their own olive crushing units (UTs), it provided a stronger motivation for the farmers and the OPAs by easing the burden of their responsibilities. The true impact of the Catalyst Fund could not be measured during the final impact evaluation, because more time needs to elapse to measure its true impacts.   

 

  • Stronger stakeholder analysis of the country and industry-specific market is needed to create better assumptions surrounding a particular project. The newly created GIEs faced difficulties commercializing their production of olive oil, despite the good quality and reasonable prices. This was discouraging for the beneficiaries, who as a result had not developed their autonomy to develop markets (marketing and commercialization). One of the underlying and often overlooked assumptions associated with this component is the fact that olive oil markets would be improved and would be fully functioning to sustain the impacts of the intervention. Improved trainings and coaching would require more in-depth analysis of the Moroccan environment. Given the depth of analysis in other economic and geopolitical analysis on Morocco’s economy, one should acknowledge the tremendous risks that exist in sustaining a business. Stakeholder analysis of who does and who would form an improved Moroccan olive-oil market, an analysis of the political and economic risks to that industry and how those risks will be met by the program intervention, could create better future interventions in improving the olive oil market in the program site.
  • There needs to be stronger linkage between the pre-production and post-production stages. The upgrading of processing units was a challenge. Although 43% of UTs had drafted a business plan by the end of the project, future projects should concentrate more on the pre-production side and help the processing units conform to laws 28-07 and 10-95. Moreover, the MAPM should continue to train and supervise the farmers and members of the OPA. Without adequate access to olive production of the best quality, the GIE and their new processing units cannot produce an oil of good quality.

 

Next Steps

MCC is dedicated to the ongoing performance and impact evaluation of the Rain-fed Olive Tree Rehabilitation Activity in order to get more accurate data that would more likely reflect the expected impacts of the implementation.  While the intermediate evaluation revealed a number of lessons learned about the activity, further analysis over a longer period (eg. two or more years after the present evaluation) is expected to reveal a better estimate of the long-term impacts of the project in terms of adoption of best practices, production, and farm revenues. Furthermore, while the Catalyst Fund processing units were not yet operational at the conclusion of the current intermediate evaluation, the indirect impacts of the Catalyst Fund’s units on treatment and control perimeters will be monitored and measured, wherever feasible.

Further independent evaluation of the Project’s activities is being carried out and is expected to be completed by 2017.

The Fruit Tree Productivity Project has also been the subject of other performance evaluations that are completed or nearing completion:

  • A performance evaluation of all components of the Fruit Tree Productivity Project
  • A cost-benefit analysis and performance evaluation of the Expansion of Rain-Fed Fruit Tree Planting Sub-activity.
Footnotes
  • 1. A perimeter is a group of neighboring parcels and represents a geographical area between 200-250 hectares.
  • 2. As noted below, continued measurement is underway, with a follow-up evaluation report expected in 2016 or 2017.