Measuring Results of the Mozambique Rural Water Supply Activity

In Context

The MCC compact with Mozambique approved a five-year investment (2008-2013) program of $506.9 million in four project areas. Of this budget amount, excluding administration and monitoring and evaluation costs, a total of $447.1 million of compact funds was disbursed in the following project areas by the September 23, 2013 compact end date: 1) Water Supply and Sanitation ($200.2m), 2) Rehabilitation/Construction of Roads ($136.8m), 3) Land Tenure Services ($39.5m), and 4) Farmer Income Support ($18.9m).

The Water Supply and Sanitation Project included four major activities: technical assistance and capacity building ($11.4m), rehabilitation and expansion of urban water supply systems ($113.5m), rehabilitation and expansion of six municipal sanitation and storm water drainage systems (62.4m), and rural water supply ($12.9m).

The Rural Water Supply Activity consisted of two components: (i) construction/re-construction of 600 improved water points in rural communities, and (ii) the mobilization of water committees to maintain water point infrastructure and provide community-based training in improved sanitation and hygiene practices.

The $12.9m Rural Water Supply Activity (2.9 percent of total compact disbursements) is the subject of this Results Summary. Results are based on an independent impact evaluation conducted through a Cooperative Agreement between the MCC Department of Policy and Evaluation’s Monitoring and Evaluation Division and Stanford University/Virginia Tech. The results of the impact evaluation are available in the final evaluation report titled “Impact Evaluation of the Mozambique Rural Water Supply Activity under a Cooperative Agreement between MCC and Stanford University”, dated March 3, 2014, to be released to the public following clearance by the MCC Disclosure Review Board in June 2014.

Pie CHart: Mozambique Compact Spending

Program Logic

Mozambique has one of the lowest levels of per-capita water consumption in the world and is far below global benchmarks. The Rural Water Supply Activity (RWSA) Project was designed to increase sustainable access to improved water supply in six of Mozambique’s poorest districts in the Northern Provinces of Nampula and Cabo Delgado. Traditional African gender norms dictate that women and girls are primarily responsible for collecting water for household use often requiring spending hours fetching water from traditional unprotected water sources. This leaves adult women less time for caring for children, the sick, and elderly or for engaging in alternative economically productive income-generating activities and less time for young adolescent females to attend school.

The RWSA installed water points to increase access to improved water sources (604 Afridev hand pumps, 8 small scale solar systems and 2 unequipped boreholes) in 614 rural communities (exceeding the end-of-compact target of 600) across the Nampula and Cabo Delgado provinces. In addition, RWSA mobilized water committees to maintain the water point infrastructure as well as provide community-based training in improved sanitation and hygiene practices. The increased access to improved water sources resulting from these investment activities was expected to reduce waterborne disease rates and decrease time spent fetching water, thus, increasing the opportunity for enhanced productivity and general well-being in the intervention areas.

Flow Chart: Program Logic

There were several key assumptions underlying the RWSA program logic during the design of the investment:

  • The time women save carrying water will be directed towards productive activities in the home, the market, or agriculture;
  • The time girls save assisting their mothers with water fetching will be directed instead towards increased school attendance;
  • Increasing access to improved water sources will increase household water consumption; and
  • With easier access to improved water, households will consume higher quality water for cooking and bathing, thus improving eventual health outcomes by reducing incidence of diarrhea and respiratory illness that result from water-borne illnesses.

 

Measuring Results

MCC uses multiple data sources for measuring results. Monitoring data are used during compact implementation. Monitoring data are typically generated by the program implementers, and specifically cover the program participants who received treatment through the compact. However, monitoring data are limited in that they cannot tell us what these program participants would have done in the absence of the MCC-funded intervention. For example, when implementers report that time to get to a non-private water source has decreased, we do not know if this has occurred because of an MCC-funded borehole or not. This is a key motivation for why MCC invests in independent impact evaluations, which establish a counterfactual in order to assess what would have happened in the absence of the MCC investment.

The following table summarizes performance on output and outcome indicators specific to the Mozambique Rural Water Supply Activity.

Monitoring Indicators Tracked During Implementation of the Rural Water Supply Activity
 
Indicators Level Actual Achieved Target Percent Complete**
Water points constructed Output 615 600 102%
People trained in hygiene and sanitary best practices Output 8,400 7,200 117%
Percentage of rural population in intervention areas with access to improved water sources Outcome 23.4* 22.1 106%
Time to get to non-private primary water source (Rural) Outcome 76 112.6 176%
Residential water consumption (Rural) Outcome 19.5 21.5 53%

*Benefiting an estimated 317,000 household members from six (6) intervention districts.

**Percent complete is calculated relative to the baseline value: the actual achieved minus baseline divided by the target minus baseline.

The average completion rate of output and outcome targets was 110.8 percent. Targets were exceeded in four of the five indicators. With regard to the “rural residential water consumption” performance indicator, it is worthy to note that although consumption only increased by 2.3 liters per capita per day, 75.8% of all Phase 2 treated community’s median total household water consumption is from an improved water source (Reference: “Impact Evaluation of the Mozambique Rural Water Supply Activity Under a Cooperative Agreement between MCC and Stanford University”, Table 7.10, pp. 59).

Evaluation Questions

The impact evaluation was designed to answer questions such as:

  • Did the installation of hand pumps improve household access to improved water?
  • Did the installation of hand pumps reduce the time households spent fetching water from a primary source?
  • Did the installation of hand pumps reduce the percentage of children under the age of 5 with reported respiratory and gastrointestinal illness?
  • Did the installation of hand pumps increase levels of household monthly income or expenditure?

 

Evaluation Results

Although most output and outcome targets for this activity were met or exceeded, the independent evalu¬ation found varied results for the RWSA outcomes. The installation of hand pumps in communities in Nampula led to significant increases in household access to improved water and reduced the time households spent fetching water from a primary source. However, as of the 2013 follow-up data collection event, the evaluation found no statistically significant health-related impacts and no relationship between the installation of hand pumps and changes in household monthly income. Communities that received a hand pump experienced an increase in median daily water consumption from improved sources of 15.1 liters per capita per day. Women and children engaged in water fetching experienced an increase in the volume of water collected ranging between 9% (3.6 liters) and 33% (10 liters). Additionally, households saw a reduction in time spent collecting water. The total time spent collecting 20 liters of water year-round fell by 42 minutes in communities that received a hand pump. Further, these households experienced a 62-minute reduction the median year-round roundtrip time to the primary source, which increased to a 129-minute reduction during the dry season. The evaluation found that time savings were used primarily for domestic activities, resting, family activities, child care, and farming. The installation of hand pumps was also associated with a 9 percent and 2 percent reduction in the percentage of children with reported respiratory and gastrointestinal illnesses, respectively. However, these decreases were not statistically significant. The evaluators conducted water quality testing to explore these results and found that the hand pumps were providing a high level of water quality at the point of collection, but at the household level, almost half of the samples of stored drinking water had levels of contamination. This implies that inadequate hygiene and water management practices obviated households’ potential health gains from the hand pumps, resulting in limited impacts on the observed illnesses. This may be attributable to a number of factors including, for example, that (i) it may be that inadequate hygiene and water management practices obviated the households’ gains in water quality from the point of collection, resulting in the limited observed impacts on child respiratory and gastrointestinal illness and/or (ii) that pathogens causing these illnesses among sample households are transmitted via exposure pathways other than and/or in addition to ingestion in water (e.g., hand to mouth contact our through food).

As of the survey end date of July 2013, the evaluation found no statistically significant relationship between the installation of hand pumps and changes in self-reported levels of monthly household income or expenditure.

 
Evaluator Principal Investigators: Jenna Davis (Stanford University) and Ralph Hall (Virginia Tech)
Evaluation Type Impact
Methodology Differences-in-differences
Exposure Period Two years: July 2011 Baseline Survey – July 2013 Follow-up Survey
Increased per-capita water consumption, increased access to improved water sources and decreased time spent to fetch water
  • Household access to an improved water source in intervention areas significantly increased
  • Consumption of improved water increased by 15.1 liters per capita per day in intervention areas
  • Women and children experienced an increase in the volume of water collected ranging between 9% (3.6 liters) and 33% (10 liters)
  • Strong evidence that total time households spent collecting 20 liters of water fell by 55 minutes when comparing treatment and comparison communities
  • Evidence that median household year-round trip time to the primary water source fell by 62 minutes and dry-season roundtrip time fell by 129 minutes
  • The installation of the hand pump can be associated with a 30% reduction in the total median time females spend collecting water each day
Increased productivity and reduced rate of waterborne diseases
  • Time savings were primarily used for domestic activities, resting, family activities/child care, and farming
  • The installation of the hand pump was associated with a statistically insignificant 9% reduction in the average percentage of children with symptoms of respiratory illness and 2% reduction in children with symptoms of gastrointestinal illness
Increased household incomes and reduced poverty rate
  • No statistically significant impacts have been detected on household income
  • No statistically significant impacts detected on poverty rate

Lessons Learned

While MCC staff are continuing to review and assess the project implementation and the impact evaluation, some areas for potential improvement are being developed.

  • Critically assess site selection decisions. Many rural water supply projects in sub-Saharan Africa are based on the installation of hand pumps at public water points. One of the key expected benefits of these investments is time savings. Surprisingly, references to rural water planning rarely provide systematic approaches to optimizing distance related benefits. MCC staff are reviewing how best to design future projects by critically assessing how MCC and MCAs: (i) identify the number and location of point sources that maximizes benefits to consumers, and (ii) determine the appropriate site location strategy, such as selection by lottery, largest unserved population, most distant unserved populations, minimization of the maximum distance, and minimization of the average distance between household and sources. Decisions related to these two points should be informed by a much better understanding of the demand – and willingness to pay for public water point services. We must also have a much better sense of the costs and particularly those costs which will be passed on to consumers. The MCC is looking to identify the degree to which benefit streams are influenced by distance to inform facility location in future projects.
  • Understand the program logic. A key finding of this evaluation is that hand pump installations have had no observable impacts on health outcomes. This may not be surprising considering what is known in the literature on how water source, storage, and consumption is related to specific water-related illnesses, such as diarrhea and respiratory infections. A critical assumption underlying the RWSA program logic was that the increase in access to an improved water source would improve the quality of water stored and used in the home. However, given water storage practices in rural Mozambique, there is a high risk of contamination within household storage containers; i.e., post water collection contamination. Rather than an assumption, this should have been a key factor considered in program design. In the future, MCC will ensure that interventions with cross-sector benefit streams, such as water and health, will require peer review to ensure critical assumptions are understood and more thoughtfully considered in program design and more thoroughly addressed in project implementation.
  • Improve Project Design and Implementation. A key finding of “no observable impacts on health outcomes” highlights the underlying importance of (i) understanding the water and sanitation knowledge, attitudes, and practices (KAP) of targeted beneficiary populations and (ii) designing/implementing effective health-based information, education, and communication (IEC) activities to complement infrastructure investments in order to produce necessary behavioral change in personal hygiene and sanitation practices. Accordingly, MCC has initiated broader inter-disciplinary collaboration among subject matter specialists (Monitoring and Evaluation, Economic Analysis, Social and Gender Analysis, Water Sector Initiatives Practice Group, and Health and Education) for conducting more detailed analyses of local community water and sanitation knowledge, attitudes, and practices as an integral part of compact development and implementation design.
  • Valuing Time Savings as a Measure of Household Income.A second key finding of “no observable impacts on household income” highlights the underlying importance of (i) better understanding of the recommended minimum amount of time that need lapse between before and after comparisons on household consumption and expenditure patterns in agricultural-based rural economies and (ii) brings into question how evaluators should assign value to time savings for the alternative use of household time on economically productive and/or non-remunerated work. Regarding the latter, the MCC Department of Policy and Evaluation is studying the possible assignment of an economic value on time savings independent of time use; equivalent to the local hourly labor market rate for unskilled labor.

 

Next Steps

There is a proposal for an MCC-funded June-July 2016 End Line Survey to assess the longer-term impacts on (i) household economic and health outcomes and (ii) the sustainability of MCC investments. The evaluation Final Report in conjunction with anonymized 2011 Baseline and 2013 Follow-up Survey data sets (household survey data and water testing data) and associated supporting technical documentation will be available on the MCC external web site for public access and use as of June 2014.