Sector Results and Learning:
Water, Sanitation, and Hygiene

Evaluations designs need to be adaptable as the situation on the ground shifts. This evaluation started as a continuous treatment approach using Generalized Propensity Score Matching (GPSM). At the midline, MCC realized that this was not a viable way to account for attribution of results. The design then pivoted to a binary treatment with Lower Ruvu households as the treatment group and Upper Ruvu households as the control. However, the Upper Ruvu households were contaminated by a new water treatment plant in that area. The design then pivoted again to a rigorous pre-post performance evaluation for always connected households and an impact evaluation for newly connected households, using Coarsened Exact Matching (CEM) to compare newly-connected with still-unconnected households. All of the changes to the design reflect the need for flexibility as the situation on the ground changes. In order to allow evaluations to be adaptable, MCC has internal processes such as annual evaluation risk reviews and the Evaluation Management Committee (EMC) to have cross-departmental input to inform evaluation decisions.

Once an evaluation methodology is agreed on in an Evaluation Design Report, the full Evaluation Management Committee must communicate continuously to ensure the design is preserved. As detailed above, the project looked to invest in new areas with its cost-savings, and chose areas that were designated “controls” for the evaluation. The evaluation team learned about this decision just in time – it was very nearly too late. Communications should be clear and continuous to ensure the “too late” scenario does not occur, and MCC investments in the agreed evaluation approach are preserved.

Test key assumptions. DRWS provided hygiene and sanitation training to communities to encourage good hygiene behaviors and provide information about the new sanitation facilities. While the household survey investigated hygiene behaviors, the evaluation did not test water samples to assess the quality at the point of source or point of consumption. As such, we are not able to test the critical assumptions that the new water sources do in fact result in increased consumption of higher quality water and that households are not inadvertently contaminating water after collecting it.

MCC should use the findings of completed program evaluations to inform its understanding of the logic and potential benefits of similar programs in development. These evaluation findings contribute to a growing body of evidence about how urban water interventions work and should help improve MCC’s design and assessment of similar interventions in the future. For example, the evaluation demonstrates that results can differ based on the customer type, e.g., new customers experienced time savings, while existing customers did not. It highlights the possibility that benefits to new customers could differ depending on whether they connect to an existing network or an entirely new network, and sheds light on other urban dynamics that should be considered, e.g., spillover benefits and the use of multiple water sources. It also demonstrates the need to plan for wastewater treatment, especially for entirely new water systems like that in Semonkong, and in cases where project benefits depend heavily on industrial water use. This planning did not happen during the design of the Lesotho project and the evaluation found that wastewater treatment was both lacking in Semonkong and a persistent constraint to industrial water use. The MCC Economist who authored the agency’s Cost Benefit Analysis Guidance for water sanitation, and hygiene, engaged on the learning coming from this evaluation and can incorporate this learning into that guidance if or when it is updated. The findings will also be available to all MCC staff and external audiences for use going forward.

There is value in conducting an infrastructure assessment as a precursor to measuring program outcomes. The evaluation’s infrastructure assessment found that Mocuba emergency works were destroyed in a flood and confirmed that the Nacala works remained incomplete. As such, MCC decided not to fund data collection evaluating those investments, since it would be impossible for those investments to demonstrate benefits. Conducting the infrastructure assessment early in the evaluation prevented MCC from funding costly data collection for infrastructure which would not reap benefits.