Measuring Results of the Tanzania Kigoma Solar Activity

The MCC compact with Tanzania was a five-year investment (2008-2013) of $694.5 million. The $11 million Kigoma Solar Activity, one component of the $199.5 million Energy Sector Project, is the subject of an independent performance evaluation summarized here. 1

The Kigoma Solar Activity installed a total of 390 solar photovoltaic (PV) systems in dispensaries, health centers, schools, village markets, and beach management units with the intent to address a range of energy needs in the Kigoma and Kasulu districts of the Kigoma Region, where energy infrastructure and access to grid electricity have been limited.
According to the MCC program logic, the activity was designed to foster sustained growth in the PV market and address the lack of sufficient electricity supply in the remote Kigoma Region, which is isolated from the national grid, by increasing access to electricity and users’ technical capacity to use and maintain PV systems, thereby increasing investment opportunities and human capital potential.
The Kigoma Solar activity was generally implemented according to plan, and uptake of the PV Solar Systems among targeted groups was widespread.
Among the lessons learned is the consideration of maintenance and repair of the PV systems and the importance of sufficient battery replacement capacity when designing future solar energy investments.

In Context

The MCC compact with Tanzania was a five-year investment (2008-2013) of $694.5 million to fund three projects: The Energy Sector Project, the Water Sector Project, and the Transport Sector Project. The Energy Sector Project, totaling $199.5 million, included three major activities: Distribution Systems Rehabilitation and Extension Activity (“T&D Activity”), Zanzibar Interconnector Activity, and the Kigoma Solar Activity. 2  The Kigoma Solar component cost $11 million and is the subject of an independent performance evaluation released by MCC in March 2017, the results of which are summarized here. This component represents 6 percent of the Energy Sector Project and 2 percent of the total compact investment. 3   The Zanzibar Interconnector Activity Evaluation Report was released in July 2015, and the T&D Activity Final Evaluation Report on March 2017.

Pie Chart: Division of Funds for Tanzania I Compact

Program Logic

The Kigoma Solar activity was designed to address a range of energy needs in the Kigoma and Kasulu districts of the Kigoma Region, where energy infrastructure and access to grid electricity have been limited because the region is served only by a mini-grid that not yet connected to the national grid. The key components of the Kigoma solar activity involved providing solar photovoltaic (PV) systems for schools, health centers, and village markets; providing resources for solar-powered night fishing systems; and selling solar systems for household and small business use with financing through local credit institutions. MCC also financed efforts to develop a market for PV systems in the Kigoma region, including marketing the solar systems and their benefits; training installers, vendors, and end users; and supporting maintenance and post-sale services.

By increasing access to electricity and users’ technical capacity to use and maintain PV systems, the Kigoma Solar activity was intended to increase the number of electricity consumers, improve the quality of service delivered, and increase consumption of solar-powered electricity thereby reducing non-solar electricity consumption, including the consumption of kerosene and diesel. This was expected to result in an increase in economic activities by both businesses and individuals and provide greater access to energy, education, and medical services.

Flow Chart of Program Logic

There were several key assumptions underlying the Kigoma Solar program logic during the design of the investment, including but not limited to:

  • Lack of grid supply was the main barrier to access to electricity among the target populations, and target groups would use PV systems if they had the means to buy them.
  • Beneficiaries would be able to buy replacement batteries and parts for the PV Systems.
  • In order for the investment to results in improved human capital outcomes, it was assumed that children would spend more hours studying after dark than they could before the intervention.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators

Level

Baseline (2013)

Actual Achieved (12/2013)

Target

Percent Complete

Capacity of household PV systems installed

Output

0

3.10

380.60

0.81%

Capacity of other PV systems installed (kWp) **

Output

0

242

241

100%

Number of PV systems installed: Dispensaries

Output

0

116.00

116.00

100.00%

Number of PV systems installed: Health centers

Output

0

14.00

14.00

100.00%

Number of PV systems installed: Vaccine Refrigerators

Output

0

130.00

130.00

100.00%

Number of PV systems installed: Secondary Schools

Output

0

45.00

45.00

100.00%

Number of PV systems installed: Village markets

Output

0

25.00

25.00

100.00%

Number of PV systems installed: Beach management units

Output

0

60.00 4

60.00

100.00%

** This indicator focuses solely on fully subsidized installations at health centers (plus vaccine refrigeration systems), dispensaries, schools, village markets, and beach management units.

Source: Closeout ITT from February 10, 2014, which includes data through December 2013, based on reporting from CAMCO.

The average completion rate of output targets is 87.6 percent. 5

Evaluation Questions

The evaluation was designed to answer questions such as:

  • How was the Kigoma solar activity implemented? Did implementation go according to the project design?
  • How do electricity use, economic activity, and human capital accumulation differ at follow-up and change over time for the targeted group selected to receive the Kigoma solar activity versus the non-targeted group?

Evaluation Results

Mathematica’s evaluation of the Kigoma Solar Activity used two rounds of data collection to compare targeted and non-targeted respondents over time in several implementation domains: 1) implementation processes and experience with the program, (2) installation of Compact-funded solar systems, and (3) service quality of solar systems. The first round of data collection was conducted by an independent consultant, contracted by MCA-T in 2013. This round of data collection could not serve as a true baseline because it was collected after most PV installations had been completed, and is thus referred to as “interim” data collection. The interim round of data collection did include retrospective questions about conditions prior to PV installation. Mathematica contracted Economic Development Initiatives (EDI) to conduct the second and final round of data collection in September 2015. Because of the small sample sizes and the purposive sampling of the non-targeted group, the estimated impacts of the Kigoma Solar Activity described in the report cannot be considered rigorous, but are helpful for assessing potential impacts of the activity.

 
Evaluator             Mathematica
Impact or Performance? Performance
Methodology    Pre-Post
Evaluation Period Implementation completed between March 2012 and May 2013

First Round of Data Collection: 2013 6

Final Round of Data Collection: September 2015

Exposure Period: 24 months

Outcomes
  • Implementation generally occurred according to plan and most of the beneficiary groups targeted received the PV systems funded by MCA-T, however none of the respondents received messages as part of the SMS marketing scheme.
  • None of the fishermen surveyed reported having had access to PV Systems, which is contrary to the monitoring data reported for the ITT.
  • PV systems face quality issues, and a lack of maintenance and repair training may limit their utility and acceptance.
  • While hours of PV electricity use increased dramatically over time for the targeted group, it appears that most of this occurred by the time of the interim survey was completed. In contrast, PV use remained low in the non-targeted group at both the retrospective baseline and interim, but increased substantially between the interim and follow-up surveys. This suggests that targeted respondents began using their PV systems shortly after receiving them, and that non-targeted respondents may have chosen to obtain PV systems through other sources in the time period between interim and follow-up data collection.
  • Liquid fuel use was lower among targeted respondents than non-targeted respondents both at interim and at follow-up, which is consistent with the hypothesis that the PV systems could substitute liquid fuels sources for energy needs.
  • Findings suggest that, in spite of training and efforts to create ownership by users, the Kigoma solar activity was not associated with any clear changes in the operations of businesses and community institutions.
  • All targeted health centers and dispensaries, and 80 percent of targeted schools, were using their MCC-provided PV systems at the time of the follow-up survey.
Objective-level Outcomes The Tanzania I Compact did not include objective-level outcomes for the Kigoma Solar Activity because it was added mid-Compact.
Effect on household income attributable to MCC The evaluation did not find any increase in income attributable to the Kigoma Solar Activity.

Lessons Learned

  • The Kigoma Solar program was designed to achieve some of the expected benefits of the canceled Malagarasi Hydropower project.  The program was intended to bring more electricity supply to the Kigoma region and find ways for communities that were not connected to the national grid, to realize benefits of alternative electricity supply modalities – in this case solar PV systems.
  • The Kigoma Solar Project included a household solar energy demonstration program as well as direct investments in electricity facilities for certain of the region’s unconnected customers.  A valuable lesson learned in this context was that outreach and marketing efforts were critical in promoting widespread uptake and should appropriately targeted to the intended beneficiaries.
  • Where lighting and heating needs were present in commercial and household settings, PV systems provide an opportunity for reducing reliance on liquid fuels used for those energy needs.
  • The sustainability of the village market and beach management unit applications depended upon organizing user groups to accept the responsibility for ownership, responsibilities, and maintenance costs.  As such, early identification and organization of the user/owner groups is important to continued operation of the systems after deployment.
  • The program did some market research on solar PV applications in the region.  The sustainability of this program depended upon attracting local distributors of related goods and services to take up solar system sales and servicing.  When planning and executing commercial systems and programs, ensuring access to resources for repair and maintenance, and having sufficient access to replacement batteries are important considerations for the sustainability of solar activity.
  • Whereas the beach management unit application was new to the Kigoma region, the health centers, dispensaries and schools either were already using solar PV systems or were part of government programs where other facilitates were using this technology.  The apparent sustainability of these latter systems, as demonstrated by the measured outcomes, indicates that solar PV systems in new settings may benefit from transferring those practices at already operating solar PV applications to the proposed new setting.
  • Program design should include target market analysis and randomized evaluation capability in data collection for feasibility studies.
  • An evaluation that measured electricity consumption for both targeted and non-targeted groups, over a longer period of time may provide answers to the slower electricity growth for target market electricity consumers.
  • Analyzing changes in energy expenditures after introduction of PV systems may inform the value of electricity in willingness to pay studies
  • Clearer documentation of the program design and what was implemented are critical to designing and implementing a comprehensive and useful evaluation.

 

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. Figures from FY16-Q4 MCC Semi-Annual Report to Congress.
  • 2. The original compact agreement did not include the Kigoma Solar Activity, as it was developed to replace the Malagarasi Hydropower and Kigoma Distribution Activity, which was canceled due to environmental risks.
  • 3. Figures from FY16-Q4 MCC Semi-Annual Report to Congress. The Malagarasi Hydropower Activity was intended to increase the energy supply in Kigoma, and the Kigoma Solar Activity was deemed a low risk alternative.
  • 4. At Follow-up none of the fishermen identified as beneficiaries reported having used a PV system. This discrepancy between the survey responses and monitoring indicators could point to data quality issues with the implementer data or misrepresentations by the fishers. For instance, they may have reverted to their old way of working and did not want to report getting the systems for one of several reasons.
  • 5. These figures are calculated using all non-evaluation indicators with targets in the Tanzania Energy Project, Kigoma Solar Activity.
  • 6. As stated above, because the first round of data collection was collected after implementation began, it cannot be considered a true baseline.