Measuring Results of the Tanzania Zanzibar Interconnector Activity

The MCC Compact with Tanzania (2008-2013) invested in the $66M Zanzibar Interconnector Activity, which replaced the undersea power cable between mainland Tanzania and Zanzibar in an effort to improve the reliability of power in Zanzibar. The expectation was that this improved reliability of power would lead to increased investment and economic activities, which would fuel economic growth and poverty reduction.  The activity successfully built 39km of submarine cable, 37km of overhead lines connecting to the cable, and a 120 MVA substation dedicated to the new cable. To evaluate the impact of the Zanzibar Interconnector Activity, the Independent Evaluator conducted a performance evaluation that used pre-post analysis comparing key outcomes before and after installation of the new cable in the hotel industry on Zanzibar’s main island.  The evaluation found that the new cable had a positive influence on the quality of the electricity, with the number of voltage fluctuations and outages decreasing significantly. However, it did not detect clear increases in revenues or investment within the hotel industry or general decreases in expenditures on energy.

In Context

The Millennium Challenge Corporation (MCC) compact with Tanzania was a five-year investment (2008- 2013) of $698 million 1  in three projects: (i) The Energy Sector Project, (ii) the Water Sector Project, and (iii) the Transport Sector Project. The Energy Sector Project included three major Activities: Distribution Systems Rehabilitation and Extension, Zanzibar Interconnector, and Kigoma Solar Power 2 .  The subject of the evaluation summarized here is the $66 million Zanzibar Interconnector Activity. This component represents 6% percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.

Pie Chart: Distribution of Funding for Tanzania I Compact

Program Logic

In an effort to promote economic growth and reduce poverty in Tanzania, MCC funded the Energy  Sector Project. The objectives of the Project were to add value to businesses and improve human capital accumulation by rehabilitating and extending transmission and distribution lines, providing solar      power systems, and replacing an undersea power cable between mainland Tanzania and Zanzibar.  The last component, the Zanzibar Interconnector Activity, was designed to address the high frequency of blackouts on the main island of Zanzibar, Unguja. The key component of the Activity was the installation of a new 39km 100 megawatt (MW) submarine cable to supplement an older 45 MW cable that was reaching its limits in terms of both capacity and lifespan. 37km of 132KV overhead lines connecting the cable to substations on both the mainland and on Unguja were also built, and 120 megavolt ampere (MVA) capacity was added to the substation in Zanzibar.  Small works were conducted with the substations on the mainland side of the cable.

MCC and Millennium Challenge Account-Tanzania (MCA-T) expected that a more reliable and better quality electricity supply would lead to the following outcomes: (1) increased investment and economic activity, (2) reduction of costs associated with having poor power quality and reliability, the most substantial of which is the cost of backup power, and (3) social gains in education and health (though these benefits were expected to be small).

 

There were two key assumptions underlying the program logic during the design of the investment:

  • In order for the new cable to improve the quality of service for Zanzibar residents, the assumption was that the quality of Zanzibar Electricity Company (ZECO) service and operations (unrelated to the cable) would not deteriorate from baseline levels.
  • In order to bring about increased investment and improved human capital accumulation, the assumption was that people have the means to purchase electric appliances that can be put to productive use or replace dirty fuels such as diesel and kerosene.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on Zanzibar Interconnector Activity output and outcome indicators specific to the evaluated program. Despite the notes explaining discrepancies between the M&E Plan and what is reported here, the details on the completion of outputs have been confirmed by the MCC and MCA-T Energy Project teams and are supported by the contractor final reports.

 

Indicators

 

Level

Baseline (2008)

Actual Achieved (09/2013)

 

Target

Percent Complete

Total KM of 132 KV constructed: Submarine cable

Output

0

39*

37*

105%*

Total KM of 132 KV constructed: Overhead lines

Output

0

37**

36.2**

102%**

Grid substation capacity installed: Zanzibar (MVA)

Output

0***

120***

120***

100%***

Technical and non-technical losses: Zanzibar

Outcome

26%

28%

20%

-34%

*The target and actual figure previously reported for this indicator in the 2013 M&E Plan and the Q20 ITT were incorrect, according to the MCA-T Energy team and subsequent verification efforts with ZECO. The previously reported target of 24.4km was stated to have come from the November 2009 Invitation for Bidders (IFB) Report, but both MCC and MCA-T Energy subsequently confirmed that the IFB estimate was 37km. The IFB report from which the original 24.4km target came cannot be found by MCC, though it is possible the report referenced 24.4 miles, which is approximately 39km. ZECO/TANESCO incorrectly reported that 37.5km of cable had been constructed by Compact close, whereas the contractor closeout report noted 39km.

Both MCA-T and ZECO state that the contractor report should be the final data source, but ZECO has not provided an explanation for the difference in their reporting. MCA-T and MCC Energy confirmed that contractors completed the full scope of work for the cable.

** The target and actual figure reported by ZECO/TANESCO in the Q20 ITT were incorrect according to the MCA-T Energy team and subsequent verification efforts with ZECO. The previously reported target of 40.7km was stated to have come from the November 2009 IFB Report, but both MCC and MCA-T Energy subsequently confirmed that the IFB estimate was 36.2km (14.4+22.2). The IFB report from which the original 40.7km target came cannot be found by MCC. ZECO/TANESCO incorrectly reported that 40.3km of cable had been constructed by Compact close, whereas the contractor closeout report noted 37km. Both MCA-T and ZECO state that the contractor report should be the final data source, but ZECO has not provided an explanation for the difference in their reporting. MCA-T and MCC Energy confirmed that contractors completed the full scope of work for the overhead lines.

*** The baseline, target, and actual figures previously reported for this indicator in the 2013 M&E Plan and the Q20 ITT were incorrect. The M&E Plan had incorrectly noted that the Zanzibar Interconnector Activity’s substation work included substation upgrades in Morogoro and Tanga, when it actually only included work on the Mtoni substation in Zanzibar. It had also incorrectly documented that the substation capacity would be added to an existing Zanzibar substation, when in reality the Compact built a new Mtoni substation that would be dedicated to the new cable; this means the baseline should have been 0. Based on discussions with MCA-T Energy, the baseline capacity for the substation was 0 MVA (not 60 MVA) and the target based on the planned 120MVA in additional capacity should have been 120 MVA (not 180 MVA). The planned addition of 120MVA has been consistent throughout. MCC was not able to verify these figures with source documents, but MCA-T Energy confirmed that contractors completed the full scope of work for the substation work.

The average completion rate of output and outcome targets is 68% percent, though MCA-T Energy confirms that the three outputs were completed as planned.  It is not clear why the technical and non- technical losses target was not met, but the MCC investment only addressed one part of technical losses that derived from the old undersea cable. One possible explanation for this result is that tariff increases on the mainland by TANESCO were not met immediately by traffic increases on Zanzibar by ZECO, which resulted in increased non-technical losses.

Evaluation Questions

As stated in the Final Evaluation Report for the Zanzibar Interconnector Activity, the evaluation was designed to answer questions such as:

  1. How did outcomes for hotels on Unguja Island change after the new submarine cable was installed compared to outcomes before cable installation?
  2. What are the hotel owners and managers’ perceptions of changes in hotel outcomes after the cable was installed?

Evaluation Results

To evaluate the impact of the Zanzibar Interconnector Activity, the Independent Evaluator conducted a performance evaluation that used pre-post analysis comparing key outcomes in the hotel industry  before and after installation of the new cable. Because the cable could potentially benefit all on the Unguja Island, a credible counterfactual for an impact evaluation could not be identified. This led to the selection of a pre-post design for the study. The evaluation focused on the hotel industry, a significant driver of Zanzibar’s economy and a large electricity consumer. This decision balanced cost and potential learning from the evaluation, and as such, data from households on the Unguja Island was not collected and consequently, changes in health and education outcomes could not be assessed.  The evaluation sample consisted of 30 hotels randomly selected from the 45 largest hotels on Unguja Island.  The hotels responded to surveys about the reliability and quality of electricity, and costs or revenues that might be affected by electricity. The surveys were administered in each month over a three month period prior to the installation of the cable and then again after the completion of the Zanzibar Interconnector Activity. Interviews with hotel staff were also conducted after the installation of the cable.

 

Evaluator

Mathematica Policy Research

Methodology

Pre-post analysis

Evaluation Period

June-August 2010 (Baseline); July-September 2014 (Endline)

The cable was inaugurated on April 10th 2013 and substation work was complete by September 2013.

Findings

The following findings from the study suggests some positive changes that occurred in the hotel sector in Zanzibar since the installation of the new submarine cable, but they do not provide evidence of impacts. Quantitative findings are based on regression-adjusted estimates:

  • The quantitative and qualitative analyses both indicate that the cable activity had a positive influence on the quality of the electricity. The number of voltage fluctuations and outages decreased significantly.
  • Hotel staff indicated that the hotel’s nonelectric energy use decreased.
  • Hotels spent more on electricity in the post-cable period, but it is not clear whether this was driven more by increased consumption or increased electricity tariffs; it is likely a combination. There is no evidence of an overall reduction in energy expenditures.
  • There was no discernible change in average hotel revenue, which may be related to factors other than the cable activity.
  • During the qualitative interviews, respondents also pointed to increased competition in the hotel sector and its negative effect on their business and revenues.
  • There was suggestive evidence of greater guest satisfaction between the pre- and post-cable period.
 

Lessons Learned

There were several key lessons learned from this evaluation for MCC and future partner countries to consider when designing and implementing similar energy infrastructure projects and evaluations:

  • The ability of evaluators to accurately measure expected results and gather information on factors that may have contributed to the manifestation of results is strengthened by greater detail from sector experts on the expected impact of the intervention on the key outcomes, i.e. a more detailed program logic or theory of change. For example, an engineering expert’s input on certain issues, such as the likely impacts of the new cable on the short term outcomes, such as reliability, and long-term outcomes such as failure of the cable, would have been helpful in designing the evaluation approach and questionnaires. It is important for MCC project teams, particularly the relevant sector experts, to clearly articulate how the proposed activities are expected to change key outcomes and to what degree.
  • The evaluation strategy had originally incorporated interrupted time series analysis of electricity consumption, outages, and quality ZECO-supplied data. However, this plan was ultimately not feasible due to concerns about ZECO data quality. The Energy Sector should consider investing in improving the data systems and quality of utilities as a complement to infrastructure investments, both to enable utilities to better manage the infrastructure and to allow for more accurate measures of results.
  • In the absence of a counterfactual, the case study approach of selecting a sector or industry that is a large contributor to the economy can provide useful insights into the influence of the project.
  • External factors that may impact outcomes in the selected sector need to be taken into consideration during the case-study design phase. For instance, in the hotel study, substantial variation in revenues that were potentially due to a high-profile tourist event and the timing of Ramadan limited the evaluation’s ability to derive conclusions with confidence about the changes in hotel revenues. This difficulty may be hard to overcome so MCC should be aware that it might be challenging to estimate changes in certain outcomes, such as revenue, in situations like this.
  • The mixed-methods approach the evaluation used (by asking respondents both quantitative and qualitative questions) is feasible and provides complementary information. For example, the qualitative data enabled the evaluation to find out about the negative effects of increased tariffs and greater competition on hotel revenues, which complemented the quantitative data on changes in revenues.
  • The timing of sector reforms and asset investment did not produce the expected outcomes. Specifically, there were compact tariff reform requirements that led to more equitable pricing of electricity for the mainland utility, TANESCO, from which ZECO purchases its power, and this resulted in higher cost electricity for ZECO. However, there were no timely changes in the Zanzibar tariff rates to compensate for ZECO’s increased cost of power, and this situation negatively impacted ZECO’s financial and operational performance. MCC has learned the importance of better coordination between policy reforms and infrastructure investments. Earlier requirements for key sector reforms and the linkage of the achieving these reforms to the size and types of asset investments considered for a compact are two mechanisms now in use to realize more of the expected outcomes.

Next Steps

There are no next steps planned for this evaluation.

Footnotes
  • 1. Approximately $3.7M of Compact M&E funds that were earmarked for endline data collection for the Tanzania project evaluations were returned to MCC to fund continuing Tanzania evaluation work. As such, MCC’s September 2014 disbursement records show that only $694.5M of the committed $698.1M were disbursed.
  • 2. The original compact agreement did not include the Kigoma Solar Activity, as it was developed to replace the Malagarasi Hydropower and Kigoma Distribution Activity, which was canceled due to environmental risks.