Summary of Findings

Measuring Results of the Lesotho Rural Water Activity

View as PDF

The MCC compact with Lesotho was a five-year investment (2008-2013) of $362.5 million. The $40.1 million Rural Water activity is the subject of an independent impact evaluation summarized here.

  • This activity aimed to increase access to improved water and sanitation in order to reduce (i) time spent collecting water and caring for the sick, (ii) incidence of water-related illness, and (iii) spending on medical care, and ultimately, increase productive activity and household incomes.
  • Over 29,000 VIP latrines and 175 water systems were constructed by the end of the Compact, with another 75 water systems completed post-compact with funding from the Government of Lesotho.
  • The program was found to have impacts on short-term outcomes including time spent collecting water and use of toilets. However, impacts on outcomes such as diarrhea prevalence and household income were not detected.
  • This evaluation underscores the importance of ensuring coordination between the various program components that are necessary to achieve results, which did not happen sufficiently in this case.
  • This evaluation is complete and there are no next steps.

In Context

The MCC compact with Lesotho was a five-year investment (2008-2013) of $362.5 million in three projects:  Health, Water, and Private Sector Development. The Water Project included four major activities, Metolong Dam, Urban and Peri-Urban Water Network, Wetlands Restoration and Conservation, and Rural Water Supply and Sanitation (Rural Water). The $40.1 million Rural Water activity is the subject of an independent impact evaluation released by MCC on January 4th 2018, the results of which are summarized here. This component represents 11 percent of the total compact. This funding was complemented by approximately $16 million from the Government of Lesotho (GOL), which has not been factored into the preceding percentage. Other components of the compact are the subject of forthcoming independent evaluations.

Pie Chart: Lesotho Compact Spending

Program Logic

Access to improved water in rural areas remained relatively static at 75-77 percent in the 25 years between 1990 and 2015, while improved sanitation in rural areas only grew from 20 to 28 percent during the same period.

The Rural Water activity aimed to construct 250 community water systems and 27,245 household-level ventilated improved pit (VIP) latrines. 1   The water systems served an average of 600 people each and consisted of either boreholes with hand pumps, gravity-fed spring catchment systems, solar-powered, or electric pumping systems.

By increasing access to improved water and sanitation, households were expected to save time in the collection of water and experience a reduction in the incidence of water-related illness, leading to a reduction in medical expenditure, and less time spent being sick and caring for the sick. Ultimately, these outcomes were expected to lead to an increase in productive activity and an increase in household income.

It is important to note that while this reflects the overall logic of the intervention, the ex-ante economic analysis conducted by MCC during compact development indicated that the costs of the activity outweighed the benefits.  This is because the baseline level of water-related illness was low as was the probability of translating time savings into income generation given overall employment levels in rural areas, and expectations for increases in non-wage production.

Flow Chart: Program Logic

For a more detailed version of the program logic, please refer to page 45 of the Rural Water Evaluation Design Report, which can be found here: https://data.mcc.gov/evaluations/index.php/catalog/137/download/613.There were several key assumptions underlying the Rural Water program logic during the design of the investment:

  • The Department of Rural Water and Sanitation (DRWS) would provide complementary training in hygiene awareness and sanitation to Village Water and Health Committees (VWHC) before construction, and would also provide aftercare training to VWHCs after construction
  • Community members would adopt improved hygiene behaviors and water would not be stored in contaminated containers
  • VWHCs would support community members’ pursuit of proper hygiene and sanitation practices
  • Water minders would be prepared to maintain water systems and would remain in their communities
  • DRWS would provide ongoing support, including trainings when necessary and perform maintenance beyond what could be completed by the village water minders

 

The program logic also implicitly assumes that appreciable decreases in water-related illness are possible and opportunities to increase productive activity are available. However, as mentioned above, the ex-ante economic analysis questioned the validity of these assumptions.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators Level Baseline (2008) Actual Achieved (09/2013) Target Percent Complete
VIP latrines built Output 0 29,352 27,245 108
Water points constructeda Output 0 175 250 70b
Water minders trained Output 0 454 500 91

Source: (Closeout ITT from June 2015, which includes data through the end of the compact, based on reporting from construction contractors and DRWS)

a “Water points constructed” is an MCC Common Indicator; for the Lesotho Compact, each water system was considered one “water point constructed” so this work would be captured by the Common Indicator.

b The remaining water systems were completed post-Compact with funding from the GOL.

 

The average completion rate of output targets is 90 percent and targets were met or exceeded in one of the three output indicators. 2

 

Evaluation Questions

The evaluation was designed to test the following hypotheses:

  • Access to improved water systems increa­ses house­­hold use of safe drink­ing water
  • Installation of a VIP increases use of improved sani­ta­tion
  • Access to improved water source re­duces time spent col­lec­ting water
  • The Rural Water activity would result in the following:
    • Reduced incidence of water related illness
    • Reduced expenditure on medical care
    • Increased school attendance
    • Time savings to households associated with better health status
    • Increased productive activity and household income

 

The evaluation covered issues related to morbidity and the probability of employment but it did not cover the following benefit streams that were modeled in the economic analysis of the program:

  • Reduction in under-5 mortality rates as result of increased water supply
  • Value of Labor Time

 

The original evaluation design did not discuss this departure from the economic analysis. However, as referenced in the revised Evaluation Design Report, MCC did not require the evaluator to link the midline evaluation results to the economic analysis: https://data.mcc.gov/evaluations/index.php/catalog/137/download/613

 

Evaluation Results

The Rural Water evaluation was based on a randomized rollout design with a 6-9 month gap planned between the end of construction of 50 (early) treatment water systems, and the start of the 50 control (i.e., delayed treatment) water systems.  However construction delays resulted in 11 or approximately 20% of the treatment systems undergoing construction concurrently with the control systems, which raised questions about the internal validity of the original evaluation design.  The evaluator addressed these concerns by using different model specifications.

 

The evaluator’s preferred specification is the instrumental variable (IV) model because it exploits the randomized treatment assignment, while accounting for the delays that occurred. The results reported below are from the IV specifications.  Results from all of the models can be found in the report.

 

 
Evaluator             NORC at the University of Chicago
Impact or Performance? Impact
Methodology    Random Assignment
Evaluation Period The activity was implemented from October 2009 through September 2013*

Baseline data collection:  December 2010

Midline data collection:  November-December 2012; April 2013

*Although the Lesotho Compact ended in September 2013, construction continued post-Compact on uncompleted works with funding provided by the GOL.

Increased access to improved water sources, sanitation, time savings ·         The program increased the likelihood of households using improved water sources by 50 percentage points, increased the percentage of household members using an improved toilet by 59 percentage points, and increased the likelihood of all household members using an improved toilet by 35 percentage points.

·         On average, the program also reduced time spent collecting water by 44 minutes per day, which represents a significant reduction from the baseline mean of 105 minutes per day.

Health and Productive Activity ·         The evaluation tested multiple outcomes related to diarrhea but no statistically-significant impacts were found.  The evaluator observes that, despite lacking statistical significance, most of the specifications show a negative coefficient for the program and diarrheal incidence, which is consistent with the program logic.

·         Consistent with the weak findings on diarrhea, there were no statistically significant impacts on either expenditures on medical care, or missed school or work days due to diarrhea.  However, this is not surprising considering the low baseline levels in these outcomes where only 10% of households had a member with diarrhea in the last two weeks, less than 1% spent money seeking medical care, and only 3% missed school or work due to diarrhea.

·         The evaluation estimated the impact of the program on 11 different labor outcomes but only found a statistically significant effect on the likelihood that women worked one hour or more in the past two weeks. The fact that women were the ones saving time on water collection as a result of the program suggests that if the program was going to have any effect on labor outcomes, those effects should be primarily observed on female labor outcomes. However, no differences were detected in the actual number of hours worked by women.

·         The loss of statistical power to detect changes could have affected the evaluation’s ability to detect small differences. Furthermore, the evaluator suggests that the limited effects on labor could be driven by low demand for labor, which is consistent with the ex-ante economic analysis.

Effect on household income attributable to MCC The evaluation did not detect an impact on household income.

 

Lessons Learned

  • Ensure the efficiency of MCC investments. The ex-ante cost-benefit analysis of this Activity estimated that costs exceeded benefits, and the evaluation supported this expectation, finding relatively small impacts on time saving and no impacts on water-borne illness. These analyses suggest that there were likely more efficient uses for MCC’s funding. Adhering to MCC’s recently-adopted investment criteria should help ensure the efficiency of MCC investments going forward. In addition, during project design, teams can use early economic analysis to establish a cap for the cost of a project given the benefits that can be expected and work to ensure the project investment does not exceed that cost ceiling without a commensurate increase in expected benefits.
  • Improve coordination between various program components to achieve results. MCC funded the construction works under the Rural Water activity while DRWS led the complementary hygiene and sanitation training with little coordination from MCA. As a result, perhaps, the training preceded construction by years in some cases and 73% of the aftercare trainings still had not been completed as of June 2016. Given the importance of these trainings to the overall logic and results of the intervention, MCC and MCA should have played a larger role in developing the content, overseeing implementation, and ensuring coordination with construction work.
  • Improve coordination between intervention and evaluation to learn what works. Although the trainings were intended to contribute to results, we cannot test the impact of them directly since they were not factored into the evaluation design. In addition, because many of the trainings preceded baseline data collection, we do not have a true baseline for those outcomes. Better planning and coordination could have created greater opportunities for learning.
  • Test key assumptions. DRWS provided hygiene and sanitation training to communities to encourage good hygiene behaviors and provide information about the new sanitation facilities. While the household survey investigated hygiene behaviors, the evaluation did not test water samples to assess the quality at the point of source or point of consumption. As such, we are not able to test the critical assumptions that the new water sources do in fact result in increased consumption of higher quality water and that households are not inadvertently contaminating water after collecting it.
  • Rigorously plan for and monitor data quality. Among the biggest challenges to this evaluation were those related to data collection and processing. The Lesotho Bureau of Statistics, which conducted the surveys, deviated from the data collection plan during both rounds of data collection, which impacted both data quality and the sample size. The following solutions might have helped with these problems: aligning incentives of the survey firm and MCC/MCA/the evaluator; technological solutions such as electronic data collection, which allows for real-time monitoring of progress, fidelity to the sample design, and data quality; and ensuring the evaluator had sufficient and continued oversight of survey operations throughout the field period to safeguard data quality.
  • Be cautious when pursuing randomized rollout designs. The Lesotho Compact attempted two randomized rollout evaluation designs and both suffered delays that threatened their internal validity and compromised their statistical power. Ultimately, the Rural Water evaluation was preserved, while the Health evaluation was not. This challenge is not unique to Lesotho. Given MCC’s five-year implementation timelines, the time taken to prepare for implementation, and the realities that often occur on the ground, completing one phase of implementation and ensuring a sufficient lag before the beginning of another phase, can be very difficult and should be considered carefully and commitment of relevant parties secured before investing significant resources in this type of design.

 

Next Steps

The evaluation design contemplated a third round of data collection to explore the trajectory of results over time. However, having reviewed the 2012/13 follow-up results, NORC and MCC agreed that an additional round of data collection was unlikely to improve our understanding of the program impacts measured at midline.

NORC proposed two alternative endline studies for MCC’s consideration: (1) assessing physical and cognitive outcomes for children who were age 0-2 years at baseline or (2) assessing the current status of supported infrastructure, hygiene-related behaviors, and community support structures. Both options represent interesting opportunities. However, since the first option goes beyond the outcomes initially targeted, coupled with the weak health effects found at follow-up, MCC did not consider it a promising investment. The second option is quite relevant but given that MCC and MCA-Lesotho had very little involvement in the complementary training on behavior change and sustaining the rural water infrastructure, the second option does not represent a direct evaluation of the MCC-funded intervention which makes it less relevant as a standalone study.

 

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. During implementation, the VIP latrine target was increased from an original target of 10,000 VIP latrines in order to ensure full coverage of households in participating communities.
  • 2. These figures are calculated using all non-evaluation indicators with targets in the Rural Water Activity.