Measuring Results of the Tanzania Distribution System Rehabilitation and Extension Activity

The MCC Compact with Tanzania was a five-year investment (2008-2013) of $694.5 million. The $124 million Distribution Systems Rehabilitation (T&D) Activity, which was part of the $199.5 million Energy Sector Project is the subject of the independent impact evaluation summarized here.

The T&D Activity built 1,277.43 kilometers (Km) of 33 and 11 kilovolt (KV) lines and 1,317.55 Km of low voltage (LV) lines with the intent to (i) improve reliability of the existing transmission and distribution system, and (ii) expand access to electricity in order to improve education, health, and economic well-being. The Financing Scheme (FS) initiative provided funding for service connections and was designed to reduce financial barriers and logistical constraints to connecting for 29 randomly selected communities, by reducing the connection fee by over 80 percent.
The MCC project logic framework posited that the combined effects of the T&D Activity and the Financing Scheme initiative would result in increased economic activity, improved standard of living, and a reduction in poverty in the long-term.
The T&D activity led to a large number of new connections, but the total number is only 10,794. This is 37 percent of the 35,000 connections expected to connect in the first year of access.
The T&D Activity increased the percentage of targeted households engaged in an income-generating activity (IGA) using grid electricity, from 9 percent to 26 percent. However, the T&D Activity had no clear impact on access to health facilities or health outcomes. There was also no clear impact on children’s time spent studying at night, but time spent watching television increased, on average, by about seven minutes per day.
The T&D Activity also increased the number of households in communities with an electrified school by 51 percent.
The FS Initiative did not have a statistically significant impact on household income, but did have an impact on total (energy and non-energy) daily consumption per capita, which is often used as another measure of economic well-being. Total daily consumption increased by 367 TZS in FS households.
Children age 5 to14 in FS communities also spent about 0.18 more hours on average (or 11 minutes) per day watching television than did children in non-FS communities. Survey respondents in FS communities also noted that they felt safer than those living in non-FS communities.
Next Steps: This evaluation is complete, and there are no future steps.

In Context

The MCC compact with Tanzania was a five-year investment (2008-2013) of $694.5 million in three projects: The Energy Sector Project, the Water Sector Project, and the Transport Sector Project. The Energy Sector Project included the Distribution Systems Rehabilitation and Extension (T&D) activity and two other Activities. In tandem with the T&D Activity, MCC funded a Financing Scheme initiative (FS) to subsidize electricity connections in a pilot group of T&D communities. The $124 million T&D Activity is the subject of the independent impact evaluation released by MCC in March 2017, the results of which are summarized here. The T&D Activity represents 18 percent of the total compact budget. The Zanzibar Interconnector Evaluation Report was released in September 2015 and the Kigoma Solar Evaluation Report was released in March 2017.

Pie Chart: Division of Funding for Tanzania I Compact.The Distribution System Rehabilitation & Extension Activity investments are equivalent to %62% of the Energy Sector Project and 18% of the total Compact investment. Figures are based on MCC obligations as of December 2017

Program Logic

The Energy Sector Project was designed to address the scarcity of reliable and high quality electricity to people in mainland Tanzania and Zanzibar,1 with the objective to increase the value added of businesses and increase income more broadly; the largest component of this was the T&D Activity. The objective of the T&D Activity was to increase grid connections and use of electricity, by building new transmission and distribution lines and expanding substation capacity in seven regions of Tanzania. In addition, the FS initiative was designed to help households overcome the upfront cost of electrification by subsidizing the household connection fee; households were required to pay for wiring costs. The T&D Activity was expected to increase access and connections to grid electricity, increase the quantity of electricity sold, and improve service quality. MCC and Millennium Challenge Account-Tanzania (MCA-T) expected that the improved access to electricity would lead to important impacts on households, businesses, and communities.

Flow Chart of Program Logic

There were several key assumptions underlying the Energy Sector program logic during the design of the investment:

  • Customers in the target populations, often rural, could afford the wiring and connection fees, which could cost well over $200, and understood the application process for connections. The Financing Scheme aimed to test this assumption by offering subsidized connections, however this was only implemented as a pilot in 29 of over 192 1  communities where the T&D lines were to be built, in the context of the impact evaluation.
  • The target populations could afford the electricity tariffs and would therefore make a permanent switch to electricity from more expensive and lower quality sources of energy. Only sustained use of lower-cost electricity would produce long-term savings and growth.
  • Use of electricity in health centers, as well as the use of electricity instead of kerosene for lighting purposes would have a positive impact on health and safety outcomes.
  • For the investment to improve education outcomes it was assumed that children in electrified households and/or schools would spend more hours studying after dark than they could before the intervention.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators*

Level

Baseline

Actual Achieved as of Sept 2013

End of Compact Target

Percent Complete

Kms of 33 and 11 KV Lines Constructed Output

0

1,277.43

1,334.16

97.62%

Kms of LV (low voltage) Lines Constructed Output

0

1,317.55

1,851.88**

71.15%

Primary substation capacity installed (MVA) Output

283.2

474.40

472.70

100.36%

* MCC has removed the outcome-level connections numbers that were reported during compact implementation because material evidence from a joint MCA-T/Mathematica data quality review suggests that the data received from TANESCO were not reliable. The data were removed on October 12, 2015. The results of the data quality review were published in late 2015.

**End of Compact Target came from bidding documents for LV line construction that were based on pre-feasibility studies.  The targets were later revised downward based on full feasibility studies, so from the project perspective, the LV line work was almost 100% complete at compact close.

 

The average completion rate of output and outcome targets is 89.71 percent; and in two of the original indicators targets were met or exceeded.  The MCA-T Energy Project Team indicated that the revised LV line target based on full feasibility studies was met by the end of compact.

 

Evaluation Questions

The evaluation was designed to answer questions such as:

  • What are the impacts of being in communities selected to receive new electricity lines through the T&D activity on indicators on energy use, economic activity, and health and safety?
  • What are the impacts of being in communities selected to receive low-cost connections to the electric grid through the FS initiative?
  • What are the impacts of being connected to the national electric grid?
  • Do impacts vary by gender, age, income, or between urban and rural areas?

 

The evaluation did not cover the benefit streams related to the rehabilitation of the substations, and corresponding impacts on existing users.  It was decided that these benefits would be captured through the monitoring data, and not the impact evaluation.

 

Evaluation Results

MCC commissioned an impact evaluation of both the T&D Activity and FS Initiative. To estimate the impacts of the T&D Activity, Mathematica used a Difference-in-Differences Method (DID) with a matched comparison group. Propensity score matching was used to select the comparison group communities and households to ensure their similarity to the intervention group communities and households.  Mathematica used a random assignment evaluation design to estimate the impacts of being in a community that was offered low-cost connections through the FS initiative. The evaluation design for the FS initiative had implications for the evaluation of the T&D activity.  The estimated overall impacts of the T&D activity also capture impacts of the low-cost connections and the outreach that took place as part of the communications campaign to engage the portion of the T&D intervention group that received the FS initiative.

The follow-up household survey was conducted from August 2015 to January 2016 with 8,899 baseline households (4,468 in the intervention group and 4,431 in the comparison group) in six regions of Tanzania and Kigoma. The final post-matching analysis sample size for the T&D evaluation was 8,897 households from 178 intervention and 182 comparison communities. For the FS evaluation, the final analysis sample was 4,911 households in 661 treatment communities and 4250 households in 250 comparison communities. The FS follow-up household survey was conducted from August 2015 to December 2015 with 828 households confirmed to be new since 2011.

 

T&D Activity
Evaluator             Mathematica Policy Research
Impact or Performance? Impact
Methodology    Difference-in-difference method with a matched comparison group
Evaluation Period The T&D Lines were completed between Oct 2012 and Dec 2013, Baseline Data Collection: Aug-Nov 2011

Final Data Collection Aug 2015-Jan 2016

Exposure Period: 20-34 months

Outcomes Connection Rates:

The T&D activity increased connection rates from 11 percent to 21 percent. Although the effects were positive, connection rates were much lower than MCC’s original projections. It is estimated that MCC achieved 31 percent of the targeted number of connections (35,000) incorporated into the cost-benefit analysis produced by MCC in 2008. By the time of the follow-up survey, an estimated total of only 10,794 new connections to MCC lines had been achieved, based on the survey sample.

Households:

  • Even though the T&D activity boosted connection rates, it had no clear impact on the overall amount of energy used by households.
  • The T&D activity increased the time children spent watching TV by about 7 minutes per day (0.12 hours).
  • The T&D activity increased the likelihood that households had a child attending a school with electricity.
  • The T&D activity had no clear impacts on health outcomes.

Businesses:

  • The T&D Activity had no statistically significant effects on the number of IGAs with which households were engaged, but did increase the percentage of IGAs that use grid electricity from 9 percent to 26 percent.

Communities:

  • The T&D activity increased perceptions of safety at night by 19 percent according to community leaders’ reports from men and women in the communities. The FS also had a positive impact—increasing perceived safety by 16 percentage points from the control mean of 47 percent.
  • Data from the community survey showed that that 53 percent of communities that benefitted from the T&D activity had an electrified school compared with 35 percent of comparison communities.
  • The T&D activity had no clear impact on outcomes such as rates of in- and out-migration, rates of formation of new households, and change of community boundaries since 2011.
Objective-level Outcomes Value Added of Businesses through the T&D Project: The evaluation did not find any statistically significant effects on the number of households operating any IGA or having a household member with a paid job
Effect on household income attributable to MCC The T&D Activity did not have a statistically significant impact on household income or non-electricity consumption. 2

 

Financing Scheme Initiative
Evaluator             Mathematica Policy Research
Impact or Performance? Impact
Methodology    Random assignment evaluation design for FS initiative.
Evaluation Period Implementation completed Aug 2013-May 2014,

Baseline Data Collection: Aug-Nov 2011

Final Data Collection: Aug 2015-Jan 2016

Exposure Period: 14-24 Months

 

Outcomes Connection Rates: FS initiative increased connection rates from 18 percent to 31 percent.  Although the effects were positive, connection rates were much lower than MCC’s original projections

Households:

  • The FS initiative increased the amount of electricity consumed by about 32 percent.
  • The FS initiative increased the amount of time that both adults and children spent watching television. Children in FS communities watched about 11 minutes per day more television than children in non-FS communities (0.18 hours).
  • The FS initiative appears to have increased health problems, related to respiration and vision, for children by about 6 or 7 percentage points, but had a positive impact on perceived safety.

Businesses:

  • The FS initiative did not have any statistically significant impacts on the operation of IGAs.

Communities:

  • Respondents in the FS community rated community safety 16 percentage points higher than the control mean of 47 percent.
Objective-level Outcomes Value Added of Businesses through the T&D Project: The FS initiative did not have any statistically significant impacts on the operation of IGAs.
Effect on household income attributable to MCC The FS Initiative did not have a statistically significant impact on household income, but total daily consumption per capita increased by 367 TZS in FS households.  However, there was no impact on non-electricity consumption. 3

 

Lessons Learned

The Final Report identifies several key factors for success in T&D construction and expansion, as well as the implementation of financing schemes.

  • MCC has not yet finalized the close-out Cost-Benefit Analysis for the T&D Activity. However in comparing the costs and economic benefits as measured in the impact evaluation, the cost of building the T&D lines was large in absolute terms as well as in comparison to the estimated impacts on household consumption and income.
  • The FS initiative was designed to help address the high cost of connecting, however the Activity achieved about 37 percent of its goal of connecting 35,000 households by the time of the follow-up survey. If the FS initiative had been implemented in all T&D communities, it seems likely that Activity would have reached nearly three quarters of its goal.
  • Electrification was expected to improve health outcomes in several ways, yet very few of these outcomes could be observed. In particular, it is unclear if electrification reduced liquid and solid fuel use, which, in turn, would improve air quality in the home. Perhaps for this reason, the evidence suggested that impacts on health are somewhat unclear.
  • Studies conducted prior to project implementation should assess the readiness of households to connect.  Household data should include housing structure type and the intended beneficiaries’ understanding of the connection process and tariff rates.  Data such as these could help inform the estimated up take rates for future electricity investments.
  • M&E conducted two Data Quality Reviews under the Tanzania I Compact, the most recent of which was implemented in 2013. At the time, TANESCO had not reported any new customers connected to compact built lines. Given the more general concerns with TANESCO data quality issues, it may have been useful to conduct the DQR at a point at which outcomes are expected to manifest, and therefore there are relevant data points to verify. In addition, the independent evaluator could provide additional oversight to data collection and the DQR.

 

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. A total of 300 communities were part of the survey; 192 for the T&D Activity Evaluation and 178 for the Kigoma Solar Activity Evaluation.
  • 2. Mathematica’s exploratory analysis of households that had connected to the grid (not just to MCC lines) showed that connecting to the grid had a positive impact on household income. Connecting to the grid resulted in a 49 percent increase in annual household income. However, this increase is not directly attributable to MCC.
  • 3. Mathematica’s exploratory analysis of households that had connected to the grid (not just to MCC lines) showed that connecting to the grid had a positive impact on household income. Connecting to the grid resulted in a 49 percent increase in annual household income. However, this increase is not directly attributable to MCC.