Summary of Findings

Measuring Interim Results of the Tanzania Water Sector Project

View as PDF

The MCC compact with Tanzania was a five-year investment (2008 – 2013) of $694.5 million. The $54.5 million Water Sector Project is the subject of an independent impact evaluation summarized here.

  • The project logic hypothesized that the WSP would lead to improvements in the quantity, reliability, and quality of the water supply that would reduce costs to households resulting from scarce or low-quality water. This would allow households to allocate more time and resources to income-generating activities, and would promote the overall Compact goal of poverty reduction through economic growth.
  • The evaluation found that the capacity of the upgraded plants did increase. However, the downstream benefits at the customer level were difficult to detect, in part, due to the changes in the water system and policies of the utilities to create more equitable access to water.
  • Lessons learned include (1) time savings that are small are unlikely to result in detectible changes in economic welfare; (2) baseline data on health and time savings are critical to understanding if the benefits have the potential to be large enough to outweigh the costs; and (3) frequent power outages are a major issue for sustainability since they damage the equipment in the plant.

A final round of data collection in Dar es Salaam took place for this evaluation in September 2017, and a final report is expected in 2018.

In Context

The MCC compact with Tanzania was a five-year investment (2008-2013) of $694.5 million to fund three projects: The Energy Sector Project, the Water Sector Project, and the Transport Sector Project.  The Water Sector Project included two major activities, Lower Ruvu Plant Expansion and Morogoro Water Supply Activities. The $54.5 million Lower Ruvu Plant Expansion and Morogoro Water Supply Activities are the subject of an independent impact evaluation released by MCC in 2017, the results of which are summarized here.  This component represents 8 percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.

Pie Chart: Tanzania Compact Expenditure

*These figures are based on MCC obligations as of August 2017

Program Logic

The WSP aimed to improve the supply and quality of water in Morogoro and Dar es Salaam through investments in water infrastructure.  The objective of the Water Sector Project (WSP) was to increase household investment in human and physical capital and reduce the prevalence of water-related diseases. The project logic hypothesized that the WSP interventions would lead to improvements in the quantity, reliability, and quality of the water that would, in turn, reduce costs to households resulting from scarce, low-quality water  (e.g., expenditures on water from vendors, time spent caring for sick members of the household). This would allow households to allocate more time and resources to income-generating activities. Increases in access and reductions in costs were also expected to increase water security.

The project logic posited that increased access would occur through increased continuity of service (via additional supply in the system) and through increased number of connections to the network. Additional supply in the system was expected to be a direct outcome of the interventions. The WSP did not invest in connections. Instead, increases in connections would be expected through: increased investment by utilities to expand coverage; unconnected households opting to connect; or re-activated connections among customers whose connections were inactive due to scarcity in supply.

Flow Chart: Program Logic

There were several key assumptions underlying the WSP program logic during the design of the investment:

  • Increased availability of treated water from the utility will improve the quality of water consumed by end users.
  • Time savings from increased water supply will result in increased time in school and other productive uses such as starting a business.

For a more detailed version of the program logic, please refer to page 23 of the Tanzania M&E Plan, which can be found here.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators* Level Baseline (2008) Actual Achieved (09/2013) Actual Achieved (12/2015) Target Percent Complete (9/2013) Percent Complete

(9/2013)

Volume of residential water consumption: Lower Ruvu (million liters per day)** Outcome 116 74.28 87.66 150 -122.70% -83.37%
Volume of residential water consumption: Morogoro (Litres/capita/day)** Outcome 98 92.22 92.02 150 -11.11% -11.50%
Volume of water produced: Morogoro Mafiga Plant ( Million litres per day)** Output 19 23.74 24.11 27 59.22% 63.84%
Volume of water produced: Morogoro Mambogo Plant (Millions liters per day)** Output 4 5.17 7.36 6 58.68% 168.21%
Volume of water produced: Lower Ruvu (Millions of liters per day)** Output 180 171.42 185.42 270 -9.53% 6.03%
Operating Cost Coverage: Lower Ruvu*** Output 108% 130.99% 125% 151% 53.47% 39.68%
Operating Cost Coverage: Morogoro*** Output 100% 95.15% 106% 115% -35.44% 36.43%

 

Source: (e.g. Closeout ITT from 09/2013, which includes data through the end of the compact, based on reporting from the utilities MOROWASA and DAWASA)

Notes:

* The WSP originally included a non-revenue water component that was ultimately dropped.  The final project design did not target reductions in non-revenue water.

** The volume indicators were not met by the end of the project because the plants were not fully operational until after the compact closed.  The interventions in Morogoro were completed and put into operation in mid-2014. Although the interventions in Dar es Salaam were completed in 2015, they could not take effect until a new transmission main funded by the Government of Tanzania was completed in March 2016. The Mambogo treatment plant is now exceeding its target because the water utility is operating it at a capacity higher than the design.

*** The WSP was not designed specifically to improve the water utilities’ operating cost coverage.  There is no documentation of how these targets were derived.

 

At the time of compact closure, the average completion rate of output targets was 38 percent and targets were met or exceeded in 1 of the 9 output indicators.  The average completion rate of outcome targets was 41.26 percent and targets were met or exceeded in 2 of the 8 outcome indicators. 1

 

By December of 2015, the average completion rate of output targets is 62.8 percent and targets were met or exceeded in 1 of the 5 output indicators.  The average completion rate of outcome targets is 36.4 percent and targets were met or exceeded in 1 of the 5 outcome indicators

 

Evaluation Questions

The evaluation was designed to answer the following questions:

IMPLEMENTATION

  • Was the MCC investment implemented according to plan?
  • What challenges were encountered? How were the challenges addressed?
  • What are the lessons learned from the design and implementation?
  • What variations in this activity might be worth considering in the future?

PRIMARY OUTCOMES:

Increased availability and quality, improved service quality

  • What is the project’s impact on water supply at the utility level?
  • Does water supply change create additional customers?
  • What is the project’s impact on the availability of, and access to, water?
  • What is the project’s impact on water quality (at the source, along distribution channel, and at the point of consumption)?
  • What are the unintended (positive or negative) results of the project?

SECONDARY Outcome

Decreased water-related illness, increased water consumption, human capital accumulation, economic activity

  • What is the project’s impact on the consumption patterns of water at the household level?
  • What is the project’s impact on health, particularly on the incidence of diarrhea for children under five?
  • Do households increase investment in physical and human capital as a result of increased access to water?
  • Do the project’s benefits and costs accrue differently to men and women (and other important sub-groups)? If so, what are the differences? What are the reasons for these differences?
  • What effects does the program have on businesses, schools, and health centers?
  • What is the likelihood that results will be sustained over time?

GOAL: Reduced poverty through economic growth

  • What is the project’s impact on poverty and income?
  • What is the cost-effectiveness or re-estimated economic rate of return (ERR) based on realized benefits & costs of the project?

The evaluation did not cover the following benefit streams that were modeled in the economic analysis of the program. More detail on this topic can be found in the Evaluation Design Report (EDR) here.  Note the EDR was updated for the endline analysis.

  • Increased water availability to enterprises
  • Household Expenditures on Water Treatment
  • Lifetime income gains due to decreased childhood stunting
  • The evaluation is not powered to detect a statistically significant effect on diarrheal disease. However, questions on diarrhea are part of the survey.

 

Evaluation Results

The evaluator used a continuous treatment approach to measure results.  The statistical models were designed to assess factors associated with outcomes and to measure the effect of water supply changes; these models do not specifically attribute any observed water supply changes to the MCC interventions.  This turned out to be a critical flaw in the evaluation.  The evaluation should have first verified that households were getting more water before assuming changes in the quantity of water consumed were a result of the project.

Further, the benefits of the water treatment plant in Dar es Salaam were contingent on the completion of the Lower Ruvu transmission main by the Government of Tanzania.  However, construction of the transmission main stalled and did not come online until March of 2016.   As such, the analysis in Dar es Salaam was essentially a second baseline.  As such, the tables below report 1-year impacts of the Morogoro investments and findings from the implementation studies of both Morogoro and Dar es Salaam investments.

 

 
Evaluator             Social Impact
Impact or Performance? Performance
Methodology    Process Evaluation
Evaluation Period The interventions in Morogoro were put into operation in mid-2014. Although the interventions in Dar es Salaam were completed in 2015, they did not take effect until a new transmission main funded by the GoT was completed in March 2016.

 

The midline household survey took place in mid-2015.  However, additional administrative data and a short process study was done in December of 2016 to inform endline analysis.  This work was included in the midline analysis.

 

The exposure period for Morogoro was 1 year.  There was no exposure for Dar es Salaam since the Lower Ruvu Transmission main was not commissioned until 2016.

Outcomes Morogoro

Mafiga Treatment Plant: The intervention aimed to increase production and improve quality. In March 2015, the plant was producing below target, and in August 2015 it was producing above capacity. The evaluation found that the plant was not treating the water properly.

 

Mambogo Treatment Plant:  Previously, the system was gravity-fed taking water from a dam and feeding it directly to the network. The new plant was in excellent condition and there was evidence of maintenance.  The evaluator identified the unreliability of power as a major issue. The plant was shut down 2-3 times per day due to power, affecting production and having long run impacts on equipment.

 

Dar es Salaam

Lower Ruvu Treatment Plant: The Lower Ruvu Plant upgrade increased capacity from 180 to 270 million liters per day (MLD).  The existing transmission line could not handle the increased water supply; a new government-funded water main was thus constructed, delaying operation of the plant.  In April 2015, the plant was operating most portions of the upgraded treatment works, and systems were in good working condition, although the new transmission main was not yet operational.

 

The greatest challenge was electricity supply. Outages occur every 1-2 days causing the plant to shut down, most often in the rainy season. Power and quality issues are causing premature failure in electronic components, disabling several backup systems.  While the plant was designed for intermittent power, the impacts appear to be more severe than anticipated.

 

In December 2016, the evaluators found that several key backup or standby components were not working. This has not affected plant operating capacity yet because the primary systems remain functional. However, given the early failure rate of electronics across several different systems at the plant, there is very little capacity in place to mitigate new problems.

Effect on household income attributable to MCC NA

 

 

 
Evaluator             Social Impact
Impact or Performance? Performance
Methodology    Pre-post
Evaluation Period Baseline data was collected in mid-2013.  The midline household survey took place in mid-2015.  However, additional administrative data and a short process study was done in December of 2016 to inform endline analysis.  This work was included in the midline analysis.

 

The exposure period for Morogoro was 1 year.  There was no exposure for Dar es Salaam since the Lower Ruvu Transmission main was not commissioned until 2016.

Outcomes Supply at the utility level

Despite system-wide supply increases, the interventions did not meet growing demand. The project increased supply in Morogoro, but the average service hours per day declined substantially, from 15 to 8 hours.  The percent of customers with 24/7 access dropped from 28 percent in 2013 to 0 percent by 2015. This was due to a policy to ensure equitable, reliable access to all customers for a shorter period of time each day.

 

Availability of, and access to, water

Access to piped water on premises:  In Morogoro, households with a connection increased from 54 to 58 percent. The two most cited reasons for not having a connection included: cost of connection and lack of coverage in the area. In Morogoro, it is not possible to attribute the increase in household connections to any one factor.

 

Primary source of drinking water:  The evaluation observed no change in Morogoro in primary drinking water sources.

 

Water Quality

Source: Neither plant in Morogoro was producing water that met WHO standards.  The utility reported that quality measures were not meeting the recommended range.   At the time of the follow-up visit in December 2016, the utility had introduced a new protocol at the treated reservoir for the Mafiga plant, which should significantly improve the quality of treated water.

 

Distribution channel & point of consumption:  41 percent of households sampled in Morogoro tested compliant on E. coli by WHO standards.

 

Objective-level Outcomes Consumption of water at the household

Household water consumption: Midline estimates of overall median per capita water consumption were 71 and 61 liters per capita per day LPCD in Morogoro in rainy and dry seasons, respectively.

 

Physical and human capital investment

Households report 7 minute of time savings per week in the rainy season and 10 minutes in the dry season.

 

It is unlikely that there will be significant economic benefits to households in these urban areas from additional water supply.

Effect on household income attributable to MCC NA

 

 

 

 
Evaluator             Social Impact
Impact or Performance? Impact
Methodology    This is a quasi-experimental impact evaluation, using generalized propensity score matching (GPSM), instrumental variables and OLS regressions.  This design utilizes a continuous treatment approach to test impacts from water supply both longitudinally (baseline to midline) and cross-sectional (midline only).  For water quality, the approach was a discrete cross-sectional treatment (midline only).

 

The design attempted to quantify the impacts of incremental changes in supply on short- and medium-term project outcomes.

Evaluation Period Baseline data was collected in mid-2013.  The midline household survey took place in mid-2015.  However, additional administrative data and a short process study was done in December of 2016 to inform endline analysis.  This work was included in the midline analysis.

 

The exposure period for Morogoro was 1 year.  There was no exposure for Dar es Salaam since the Lower Ruvu Transmission main was not commissioned until 2016.

Objective-level Outcomes Consumption of water at the household

Household water consumption: Each hour of increased supply is associated with between 0.8 and 8 liters per day of additional consumption per household member. The evaluation did not find that better quality water resulted in changes in water consumption.

 

Water treatment: The evaluation was not able to find a consistent relationship between changes in water supplied and treatment of primary drinking water.

 

Household water expenditures:  The project logic posits that increased supply from the distribution system will decrease household water expenditures on outside sources.  The evaluation did not find a consistent relationship between changes in supply and household expenditure.

 

Physical and human capital investment

Household water collection:  The project logic posits that an increase in supply leads to a decrease in consumption of sources outside the household and a reduction in time spent hauling water. The evaluation found weak evidence that improved supply leads to increased time savings.   Further, the evaluation found that the amount of time spent hauling water is quite moderate, a maximum of about two hours per week both before and after the project.

 

It is unlikely that there will be significant economic benefits to households in these urban areas from additional water supply.

Effect on household income attributable to MCC The evaluation measured time savings and water expenditures.  Time savings improved marginally between baseline and midline while water expenditures increased between baseline and midline.

 

Lessons Learned

Project

  • The main health benefit stream in the CBA is derived from stunting and reduced stunting is assumed from lower rates of under 5 diarrhea. However, the levels of diarrhea were not very high to begin with and actually slightly increased (not statistically significant) at midline in both Dar es Salaam and Morogoro.  The causal links that trace the project to reduced stunting is tenuous at best.
  • Sustainability is an issue. Frequent power outages damage the equipment resulting in revenue losses.  These revenue losses then make maintenance more difficult creating a vicious circle.  To ensure the sustainability of infrastructure investments, there needs to be adequate consideration around training staff to operate and maintain the equipment and replacing of broken parts.  MCC is addressing this lesson through the new compact development guidance which focuses more on problem analysis.

 

Evaluation

  • Water network monitoring is a critical complement to household surveys. MCC should have been looking at water supply increases to the system on a consistent basis to better understand how the water was getting to the beneficiaries, if at all, rather than relying solely on household surveys.  If MCC had understood MORUWASA’s water rationing changes ahead of launching a household survey, we would have been able to better allocate evaluation resources.  MCC is addressing this lesson by placing a greater emphasis on monitoring in throughout the compact lifecycle.
  • This evaluation planned to employ a continuous treatment approach to understand how different levels of water supply affect behavior at the household level. This design relied on a critical assumption that increasing water production at water treatment plants would automatically result in more water per customer.  This assumption was not realistic and did not hold, which posed a challenge for being able to attribute the results of the continuous treatment analysis to MCC’s investment.  Future applications of this methodology must carefully consider attribution and validation of outputs and intermediate outcomes.

 

Next Steps

A final round of data collection in Dar es Salaam took place in September 2017.  This report will serve as an endline for the activities in Morogoro.  After a thorough review, MCC decided that the cost of another household survey in Morogoro outweighed the additional learning that was expected for endline.

In Dar es Salaam, given the issues with attribution of the continuous treatment approach, the evaluation design shifted for endline.  The revised EDR for endline can be found here.

A final evaluation report is expected to be released to the public in late 20

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets in the Tanzania Water Sector Project.