Measuring Results of the Namibia Indigenous Natural Products Activity

The MCC compact with Namibia was a five-year investment (2009-2014) of $304.5 million. The $7.6 million Indigenous Natural Product (INP) Activity is the subject of an independent performance evaluation summarized here.

  • The INP activity provided training and assistance intended to increase the volume, quality, and sales of INPs, and ultimately increase INP income and overall household income for INP harvesters.
  • Over 9,000 harvesters were trained, including over 5,000 trained in sustainable techniques.
  • The evaluation found that sales and INP-related income increased for Marula harvesters only, but did not detect an increase in overall household income for harvesters of any INPs.
  • This evaluation is complete and there are no planned next steps.

In Context

The MCC compact with Namibia was a five-year investment (2009-2014) of $304.5 million in three projects:  the Education Project, the Tourism Project, and the Agriculture Project.  The Agriculture Project included three major activities, the Land Access and Management activity, the Livestock Support activity, and the Indigenous Natural Products (INP) Development activity.  The INP activity consisted of three components:  Support to Producer and Processor Organizations (PPOs), Provision of the INP Innovation Fund, and Delivery of Market Information. The $7.6 million INP activity is the subject of an independent performance evaluation released by MCC in 6/13/2017, the results of which are summarized here. This component represents 2.5 percent of the total Namibia compact. Other components of the compact are the subject of independent evaluations still underway when this summary was published.

Pie Chart: Namibia Compact Spending

These figures are based on MCC obligations as of August 2015.

Program Logic

The INP activity provided training and assistance to improve harvesting practices of individuals and the operational and business capacity of PPO managers; supported research, testing and application of new innovations and services considered critical to the INP industry’s immediate, short‐term and long‐term competitiveness; and supported marketing efforts. These three components aimed to increase the volume, quality, and sales of INPs, and ultimately increase the INP income and overall household income for INP harvesters. Given that INP harvesters are traditionally women, this intervention aimed to improve livelihoods for women, in particular.

Program Logic: Indigenous Natural Products

There were several key assumptions underlying the INP activity program logic during the design of the investment:

  • Incomes of INP harvesters can be increased through demand-inducing interventions that:
    • Improve supply and quality
    • Support new and innovative harvesting and processing techniques (to further improve supply and quality)
    • Identify new products
  • Increases in demand for INPs can be managed in an environmentally sustainable manner.

 

For a more detailed version of the program logic, please refer to page 12 of Annex 4 of the Namibia M&E Plan, which can be found here: https://assets.mcc.gov/documents/ME_Plan_-_NAM_-_V7_-_Jul14.pdf.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes program performance in terms of output and outcome indicators specific to the evaluated program.

 

Indicators Level Baseline (2009) Actual Achieved (09/2014) Target Percent Complete
PPOs trained in organizational management Output 0 61 60 102%
PPOs trained in business and marketing principles Output 0 60 60 100%
PPOs that have developed business plan Output 0 48 48 100%
PPOs with resource management/monitoring plans for environmentally fragile INPs Output 0 30 30 100%
PPOs certified Output 0 3 2 150%
Farmers trained Output 0 9,238 7,000 132%
INP producers who have been trained in sustainable harvesting techniques Output 0 5,272 1,250 422%(a)
Value-added of INP processing Outcome 296,029 1,341,878 355,235 1766%(b)
Payment to producers from INP sales Outcome 1,179,319 3,802,523 4,174,319 88%

Source: (e.g. December 2014 ITT, based on reporting from MCA-Namibia, the Indigenous Natural Products PPO Consultant, and the Conservancy Support/Indigenous Natural Product household survey)

(a) Trainees mainly represent Devil’s Claw harvesters; as reflected in the percent complete, significantly more Devil’s Claw harvesters participated in the program and training than initially expected.

(b) Targets were set conservatively due to limited information at the beginning of the intervention.

 

The average completion rate of output targets is 114 percent (not counting INP producers who have been trained in sustainable harvesting techniques, which had a flawed target and an uninformative percent complete figure) and targets were met or exceeded in 7 of the 7 output indicators.  The average completion rate of outcome targets is 88 percent (not counting Value added of INP processing, which had a flawed target and an uninformative percent complete figure) and targets were met or exceeded in two of the two outcome indicators with targets.  1

 

Evaluation Questions

The evaluation was designed to answer questions such as:

  • Do the technical assistance package and the small grants increase the quantity and quality harvested and/or processed by recipients?
  • What is the uptake rate and effect of the practices and techniques introduced as part of the technical assistance on recipient harvesters?
  • Did the technical assistance improve the PPOs’ organizational capacity to manage the business and income/revenue?
  • How sustainable are the results in terms of increased production, sales and income? E.g., market chain (are there long-term buyer contracts in place, are the institutions functional and independent).
  • Did the composition and level of household incomes change (more income sources, more diversification, and higher income)?
  • What is the perceived impact on household gender relationships from the intervention among recipients?

 

Evaluation Results

This evaluation relied on a model-based approach complemented by qualitative data. The credibility of these findings hinge on acceptable assumptions underlying the model and consideration of all reasonable confounders.

 

In addition to the results summarized below, this performance evaluation found that the intervention was correlated with an increase in sales and INP-related income for Marula harvesters only. However, the evaluation did not detect an increase in overall household income for harvesters of any INPs.

 

 
Evaluator             NORC at the University of Chicago
Impact or Performance? Performance
Methodology    Pre-post
Evaluation Period 2011-2014
Outcomes ·         Qualitative reports indicate that the INP sector is more organized now than before the activity and PPO capacity has improved along some dimensions

·         However, some management trainings were considered too complex and a general consensus is that PPOs will continue to require external support, which raises sustainability concerns

·         Harvesters report applying methods taught in INP trainings

·         Qualitative findings on gender relationships are mixed, with some respondents reporting that women have more control over household income, while others report that nothing has changed

Objective-level Outcomes ·         Qualitative reports indicate that the quality of INPs has increased

·         Quantitative data indicate that the activity was associated with increased quantity harvested and sales for Marula only, with 1 additional training session associated with an 8kg increase in sales

Effect on household income attributable to MCC ·         N/A

 

Lessons Learned

  • It is important to balance supply and demand. For INPs that were in high demand, such as Ximenia, ensuring a consistent supply proved challenging; for INPs which were in abundant supply, creating higher demand nationally and internationally was difficult.  The evaluation noted that the intervention emphasized the supply side of the value chain more than the demand side, so perhaps more could have been done to address the imbalance.  Future interventions that focus on production of agricultural or other goods, should consider both supply and demand since getting the balance right is key for achieving and sustaining results.
  • Analysis plans should inform data collection. As noted by the evaluators in their revised Evaluation Design Report, the baseline household survey occurred before the evaluation team was hired. Although driven by a sense of urgency to capture a baseline, failing to fully anticipate the evaluation’s data needs ultimately served to limit the evaluation.  Therefore, it is critical to ensure that evaluators are hired and able to guide the sampling methodology and instrument design to collect the data necessary to complete desired analysis.

 

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets in the Namibia Indigenous Natural Product Activity.