Measuring Results of the El Salvador Water and Sanitation Sub-Activity

The MCC Compact with El Salvador was a five-year investment (2007-2012) of $449.6 million. The $16.6 million Water and Sanitation Sub-Activity is the subject of an independent impact evaluation summarized here.

  • The Water and Sanitation Sub-Activity provided new or improved piped water to 7,634 households and latrines for households that previously did not have access to improved sanitation. The interventions were expected to result in an increase in the quantity and quality of water available for consumption, increased consumption, a reduction in the time and cost spent on seeking or purchasing water, and a reduction in morbidity from water-related illnesses. In turn, reductions in water-related disease and time spent collecting water were expected to lead to reduced expenditures on health care, increased attendance at school and work, and more time for productive activities. It was expected that as a result, household income would increase.
  • Positive effects were found on the ownership and use of improved water and sanitation services, as well as on the reliability of service. The time households spent on gathering water and doing laundry outside the home decreased, primarily benefiting women and female children. No effects were detected on water consumption, diarrhea incidence, or school enrollment and attendance – which was not surprising as water consumption and school enrollment were high in the baseline and diarrhea incidence in the Northern Zone substantially decreased prior to program implementation. There was no evidence of increased time spent on income-earning activities, including wage labor or a household business. A small increase in household expenditure was found, but no effect on household income.
  • Through this evaluation it was learned that providing private water connections for households can be effective at freeing up time for household members, but additional interventions may be necessary to turn that time into productive time for the household. In addition, evaluations should be designed to answer “why not” if impacts do not occur as expected.
  • This evaluation is complete and there are no planned next steps.

In Context

The MCC Compact with El Salvador was a five-year investment (2007-2012) of $449.6 million in three projects: the Connectivity Project, the Human Development Project and the Productive Development Project. The Human Development Project included two major activities, the Education and Training Activity and the Community Development Activity. The Community Development Activity consisted of three Sub-Activities: Rural Electrification, Community Infrastructure, and Water and Sanitation. The

$16.6 million Water and Sanitation Sub-Activity is the subject of an independent impact evaluation released by MCC in January 2017, the results of which are summarized here. This component represents 3.7 percent of the total Compact. Other components of the Compact are the subject of already published and forthcoming independent evaluations.

Pie Chart: Distribution of Compact Spending

*These figures are based on MCC obligations as of March 2016

 

Program Logic

The Water and Sanitation Sub-Activity was designed to enhance access to water systems and to improve sanitation services to the poorest inhabitants in the Northern Zone of El Salvador. It was designed to provide piped water or (in a few cases) public taps for households that previously did not have access to this level of service and latrines to all water project participants who did not already have improved sanitation. In addition, communities were provided technical assistance to ensure appropriate system maintenance, promote the sustainability of the infrastructure projects, and community education related to appropriate health and sanitation practices.

The interventions were expected to result in an increase in the quantity and quality of water available for consumption, increased consumption, a reduction in the time and cost spent on seeking or purchasing water, and a reduction in morbidity from water-related illnesses. In turn, reductions in water-related disease and time spent collecting water were expected to lead to reduced expenditures on health care, increased attendance at school and work, and more time for productive activities. It was expected that as a result, household income would increase.

Flow Chart: Program Logic

 

There were several key assumptions underlying the Water and Sanitation Sub-Activity program logic during the design of the investment:

Time savings would be significant and would be used for productive activities (e.g. studying, work).
Cost savings would be significant because households were buying expensive water from trucks before the project.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

 

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators

Level

Baseline (2007)

Actual Achieved (Dec 2012)

Target

Percent Complete

Access to improved water supply in the entire Northern Zone (percentage)

 

Outcome

 

79

 

83.1

 

83

 

102.4

Access to improved sanitation in the entire Northern Zone (percentage)

 

Outcome

 

82

 

87.72

 

83

 

572

Households with access to improved sanitation

Output

0

7,169

1,967

364.5

Households with access to improved water supply

Output

0

7,634

6,935

110.1

People trained in hygiene and sanitary best practices

Output

0

2,406

1,967

122.3

 

Source: Closeout ITT from December 2012, which includes data through the end of the Compact.

 

The average completion rate of output targets is 199 percent and targets were met or exceeded in 3 of the 3 output indicators.  The average completion rate of outcome targets is 337 percent and targets were met or exceeded in 2 of the 2 outcome indicators. 1

 

Evaluation Questions

The evaluation was designed to answer the following questions:

Service, Use and Sustainability

  • Were the water and sanitation projects implemented according to plan?
  • Are the results from the activity expected to be sustained over time?
  • Did the MCC investment reach intended/unintended beneficiaries?

 

Coping Costs – Time and Money

  • Do water and sanitation interventions reduce coping costs? What factors might explain the impact (or lack of impact) in this area?
  • Do they reduce cash expenditures on water and on sanitation services? What factors might explain the impact (or lack of impact) in this area?

 

Gender and Social Exclusion

  • Do the effect on health, education and access of water and sanitation interventions differ by gender or by expenditure levels (initial conditions)?
  • What factors (hygiene behavior, source and household-level water quality, household source choice) might explain the impact (or lack of impact) in a specific subpopulation?

 

Education

  • Do water and sanitation interventions increase school enrollment among children between the ages of seven and 12? And children between the ages of six and 18? What factors might explain the impact (or lack of impact) in this area?
  • Do water and sanitation interventions increase school attendance among children between the ages of seven and 12? And children between the ages of six to 18? What factors might explain the impact (or lack of impact) in this area?

Health

  • Do water and sanitation interventions reduce incidence of diarrheal illness?
  • What factors (hygiene behavior, source and household-level water quality, household source choice) might explain the impact (or lack of impact) in this area?

 

Economic Well-Being – Expenditure and Income

  • Do water and sanitation infrastructure investments increase household expenditure or income? What factors might explain the impact (or lack of impact) in this area?
  • What are the consequences of water and sanitation investments for expenditure patterns?

 

Evaluation Results

The Water and Sanitation Sub-Activity impact evaluation used propensity score matching along with difference in differences to estimate the impact of the intervention on key outcomes. A unique survey was designed with a sample size of about 3,222 households in “treatment” and “comparison” areas to collect data on these key outcomes at baseline (2011) and two follow-up periods (2012 and 2013). The results are presented below.

Evaluator

Social Impact with the University of Maryland

Impact or Performance?

Impact

Methodology

Difference-in-Difference

Evaluation Period

Program implementation: March 2011 to July 2012 Baseline data collection: March – April 2011 Interim data collection: March – April 2012

Final data collection: March – April 2013

Exposure period after final data collection: Six months to two years

Outcomes

Service, Use and Sustainability

  • The probability of a household in a treated segment having a household tap increased by 25 percentage points.
  • The availability of water from household taps increased between 11 and 17.5 hours each week.
  • The probability of the household using tap water for drinking and cooking increased by 30 percentage points.
  • The probability of having improved private sanitation increased slightly, by about 3 percentage points. Children between the ages of three to six years old were 16 percent more likely to use sanitation facilities frequently.
  • There was no effect detected on water consumption.
 

Coping Costs – Time and Money

  • The probability of a household collecting water decreased by 20 percentage points and the time spent collecting water decreased by 3 hours per week.
  • The probability of a household doing laundry outside of the home decreased by 12 percentage points. For the families who did laundry outside the home, the time spent was reduced by 3 hours per week.
  • On average, expenditures on water increased by about $2 per month. There was a decrease of $5 per month on water from trucks and of $1.75 from other sources. There was also a decrease of $0.25 per month in chlorine expense for water treatment.

Gender and Social Exclusion

  • The reduction in time spent doing laundry outside the home benefitted primarily women.
  • Female children were 4 to 7 percent less likely to spend time doing laundry outside of the home.

Objective

Education

  • There was no evidence of increased school enrollment or attendance for school-aged children. This was not surprising given baseline levels of enrollment were 95 percent for children between the ages of 7 and 12.

Health

  • There was an increase in residual chlorine levels for household drinking water, but the level of chlorine was lower than the recommendable amount.
  • There was no evidence of a reduction in the diarrhea rate among children under 5. This was not surprising as diarrhea rates decreased substantially before the project was implemented.

Productive Activities

  • There was no evidence of increased time spent on income- earning activities, including wage labor or a household business.

Effect on household income attributable to MCC

Economic Well-Being – Expenditure and Income

  • There was a marginally significant increase in household expenditures of 5 percent, excluding water expenses.
  • No evidence of increased household income.

 

Lessons Learned

Providing private water connections for households can be effective at freeing up time for household members, but additional interventions may be necessary to turn that time into productive time for the household. The evaluation shows that households have freed up a significant amount of time from no longer gathering water or doing laundry outside the home; however, the saved time did not translate into increased income for the household. One possible theory is that there are few opportunities for households to use that time for productive activities. If that is the case, additional interventions aimed at income-generation for the household may be necessary.

Intermediate indicators should be included in evaluations to track the theory of change. It should be noted that even though no change in diarrhea was detected, an intermediate indicator (residual chlorine levels) did show an improvement. The longer the chain of events between the intervention and the measured outcome, the harder it can be to determine whether there was an impact. Using indicators (e.g. residual chlorine) that link intervention (water infrastructure) to outcome (diarrhea) should be replicated in future evaluations to understand how far down the chain the effects of the intervention were felt.

Evaluations should be designed to answer “why not” if impacts do not occur as expected. The evaluation leaves some questions unanswered, such as “why is there still water contamination at the household?” and “why are households not using their saved time for income-producing activities?” Qualitative research methods should be used for future evaluations to try to answer these questions in order to provide more learning for MCC, our partner countries, and other investors in the sector.

High costs per beneficiary can kill even the best projects. The economic rate of return for the El Salvador Water and Sanitation Sub-Activity was estimated as 3.6% at the close of the Compact, due largely to low population densities and high costs. While the execution of water, sanitation, and hygiene projects can be improved using the above lessons, it must be kept in mind that if we target expensive or low density regions, the benefits are unlikely to justify the project.

MCC has learned from these lessons and is working on incorporating them into how Compacts are designed, implemented, and evaluated in these ways:

Application Procedure to “Facility”: The selection of projects may have been suboptimal due to the design of the facility including but not limited to the definition of the problem that the facility was intended to solve, the planning approach that formed the basis for eligibility, the use of appropriate selection criteria and competition, and the appropriate provision of technical support. In addition, the evaluation was not designed to elucidate implementation effectiveness of the facility structure towards achieving overarching goals. To that end, MCC has        developed improved guidelines for the use of infrastructure facilities, specifically to select projects that will yield the intended outcomes.

Project Design and Implementation: MCC has learned to focus on the cost side of the ledger to better identify less costly solutions. Often times a key challenge is the use of quality and quantity standards that may not be optimal in a local context; essentially this leads to over design and higher than optimal costs.  As a result of this and similar studies, MCC is being more aggressive about seeking more cost-effective solutions.

Health and Social Benefits: As a result of this and similar studies, MCC understands that delivery of health (reduction in diarrhea, stunting, etc.) and social (increased school attendance) benefits require complex interventions that involve multiple inputs in order to effect change. This implies that theories of change and the design of interventions need to take a larger systems view, which MCC is doing in more recent Compacts.

Evaluation Design and Implementation: Evaluation scopes of work are including more emphasis on documenting implementation, qualitative data collection, and answering “why” when expected results do not materialize.

 

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets in the Water and Sanitation Sub- Activity.