Measuring Results of the Ghana I Water and Sanitation Sub-Activity

The MCC compact with Ghana was a five-year investment (2007-2012) of $547 million. The $15 million Water and Sanitation Sub-Activity, part of the Rural Development Project’s Community Services Activity, is the subject of an independent impact evaluation summarized here.

  • Program Logic: The theory of change behind the water activity is that by improving water systems in districts participating in the Compact, the households’ economic productivity and income will increase. Due to reduced time fetching water and caring for the sick, community members would have more time to devote to economic activities.

 

  • Evaluation Results: The evaluation results showed that the intervention reduced time spent fetching water by 13 minutes and diarrheal prevalence by 6 percentage points. The evaluation did not find a statistically significant impact on household welfare in terms of income or possession of durable goods.

 

  • Lessons Learned: (1) Projects should be designed to address the root cause of a clear and evidence-based problem. (2) There should be a strong understanding of the local context during project design and implementation.  (3) It is important to have a good sense of the counterfactual before the project starts and to monitor the counterfactual assumptions during implementation. (4) Qualitative studies should be better focused and include specific evaluation questions.

 

  • Next Steps: The endline qualitative and quantitative studies represent the final piece of the Water and Sanitation Sub-activity evaluation. There will be no further steps.

In Context

The MCC compact with Ghana was a five-year investment (2007-2012) of $547 million in 3 projects:  the Agriculture Project, the Rural Development Project, and the Transportation Project. The Rural Development Project included three major activities, the Community Services Activity, the Financial Services Activity, and the Procurement Capacity Activity. The Community Services Activity consisted of three sub-activities: Water and Sanitation, Education, and Electricity. Estimated at approximately $15 million 1 of the $44 million Community Services Activity, the Water and Sanitation Sub-Activity is the subject of an independent impact evaluation.

Pie Chart; Ghana Compact Obligations

The Community Services Activity is equivalent to 58% of the Rural Development Project and 8% of the total Compact investment. 2

Program Logic

The Community Services Activity was intended to complement and enhance the sustainability of the Agriculture Project by providing the necessary infrastructure to improve health of communities through water systems, to enhance skill development through access to education, and to facilitate small-scale postharvest processing of agricultural products through electrification, and by providing capacity building support to local government institutions. As part of Community Services, the Water and Sanitation Sub-Activity was designed to provide improved water systems to 137 selected communities in the intervention area in Ghana. Over the course of the project, a total of 392 water points were constructed and rehabilitated, including boreholes, small town water systems, and pipe extensions. In improving the quality of water, the sub-activity aimed to reduce the prevalence of waterborne illness and the time spent caring for the sick. Additionally, time formerly reserved for collecting water could be reallocated to income producing activities, particularly in agriculture. By improving water systems in districts participating in the Compact, the program’s goal was to increase the households’ economic productivity and income.

Flow Chart: Program Logic

There were several key assumptions underlying the Water and Sanitation program logic during the design of the investment:

  • The time women save carrying water will be directed towards productive activities in the home, market, or agriculture.
  • The time children save assisting their families with water collection will be directed instead towards increased school attendance.
  • Increasing access to improved water sources will increase household water consumption.
  • Households will be willing to pay to obtain water from improved and more convenient access points.
  • With easier access to improved water, households will consume higher quality water for cooking and bathing, thus improving eventual health outcomes by reducing the incidence of diarrhea and guinea worm that result from water-borne illnesses.

 

For a more detailed version of the program logic, please refer to page 4 of the Ghana M&E Plan, which can be found here.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators Level Baseline Actual Achieved (07/2012) Target Percent Complete
Number of people affected by water and sanitation facilities sub-activity Output 0 153,853 129,840 118%
Number of people trained in hygiene and sanitary best practices Output 0 778 1,661 47%
Number of women trained in hygiene and sanitary best practices Output 0 246 N/A  N/A
Number of water points constructed Output 0 392 377 104%
Distance to collect water (Meters) Outcome 1190 522 500 97%
Time to collect water (Minutes) Outcome 43.67 31.76 30 87%
Number of reported guinea worm cases  in MiDA Districts Outcome 252 0 10 104%
Volume of domestic water consumption (Liters per capita per day) Outcome 15 36 20 420%
Number of households with access to improved water supply Outcome 0 27,407 21,800 126%

Source: (July 2012 ITT, based on reporting from the MiDA Data Collection Support Consultant, the Ghana Guinea Worm Eradication Program (GGWEP), the Ghana Statistical Service (GSS), and the Community Services PMSC)

 

The average completion rate of output targets is 90 percent and targets were met or exceeded in two of the three output indicators.  The average completion rate of outcome targets is 167 percent and targets were met or exceeded in four 3 of the five outcome indicators. 4

Evaluation Questions

The quantitative part of the evaluation was designed to answer the following questions:

Impacts on Health

  • What was the impact of the water component in the incidence of diarrhea among children under five?
  • What was the impact on the incidence of diarrhea when households treated the water for consumption?
  • Was the reported incidence of diarrhea affected by how the question was asked?
  • Did the gender of the respondent affect the measurement of the incidence of diarrhea among children under five?
  • What was the impact of the intervention on hand-washing behavior?

Time savings

  • What was the impact of the water component on the distance traveled and the time households devoted to collecting water?
  • Did the distance between the water sources and household affect decisions about where to fetch water from?
  • How did these impacts differ by gender?

Quantity of water consumed

  • What was the impact of the investment on the quantity of water consumed by households for domestic purposes?
  • Did the gender of the respondent affect the measurement of the quantity of water consumed by households?
  • Was there a relationship between the distance from the household to the source and the quantity of water used?
  • What factors affected households’ decisions about whether or not to fetch water from an improved source?
  • What was the impact of the water component on the quality of water consumed?

Price of Water

  • What was the impact of the investment on the price that households paid for drinking water?
  • How did the pricing of water affect the quantity of water used in households?
  • What was the impact of the intervention on whether or not people were willing to pay for water?
  • Did the investment decrease the price that households pay for drinking water?

Household welfare

  • What was the impact of the investment on household income and households’ consumer and durable goods?

 

In addition to these areas, the report also explores two more areas: spillover effects and sustainability of the intervention with the following questions:

  • Spillover effects: Did the investment reach unintended beneficiaries?
  • Sustainability: Were the benefits sustainable? Were the water points still in use?

 

The qualitative study was designed to inform and refine the end-line survey instrument used for the impact evaluation and provide supplementary qualitative information that could be used to interpret and use the quantitative evaluation results.  As such the study was designed to explore:

  • Community Involvement,
  • Direct and Indirect impacts,
  • Sustainability, and
  • Gender

The economic analysis identified time savings from reduced time collecting water and reduced water borne diseases as the main benefit streams.  The evaluation covers both of these benefit streams. More detail on this topic can be found in the Evaluation Design Report here.

Evaluation Results 

This evaluation integrates both qualitative and quantitative methods to understand not only the direction and size of the impact but also the underlying mechanisms, processes, and channels through which the impact was generated.  This evaluation follows the quasi-experimental propensity-score matching design, gathering longitudinal data on the same households at baseline and end-line allowing the evaluator to control for fixed effects at the household level.  The evaluation surveyed 100 communities (50 treatment/ 50 control), with 12 household level surveys in each community.

Quantitative
Evaluator             The University of Notre Dame, Notre Dame Initiative for Global Development (NDIGD)
Impact or Performance? Impact
Methodology    Propensity Score Matching
Evaluation Period All of the water points were completed by January 2012. The evaluation assumes that by 2014 the benefits of water activity might be visible and measurable in the communities. Quantitative data collection took place in February/March 2015.  Water testing took place in May 2015.
Outcomes Health:

The evaluation found a statistically significant reduction in diarrhea incidence in children under five of 6 percentage points.   The evaluation did not find that the intervention had an effect on hand-washing as a result of the hygiene and sanitation component of the activity, though females overall reported always washing hands before eating and cooking and after going to the toilet.

 

Time and distance:

The evaluation found that the intervention reduced time spent collecting water by 13 minutes per day, and there was a three-minute reduction in travel time to collect water. No difference was detected in distance to closest water source in kilometers between treatment and comparison communities.  The evaluation states that the discrepancy between the results of less time being spent collecting water but no difference in distance are likely coming from two things: (1) people in treatment communities perceive that they spending less time collecting water, and (2) people might have actually preferred a particular water source regardless of the distance.

 

Water Consumption:

The evaluation did not find any statistically significant effect on the quantity of water consumed as a result of the intervention. The single significant factor in choice of water to drink was taste, and improved sources did not necessarily taste better than unimproved sources.

 

Price of Water

The evaluation did not find the intervention to have a statistically significant effect on the price paid for water.

 

Spillover Effects

The evaluation found that people from neighboring communities not targeted by the invention collected water from 48% of the intervention communities.

 

Sustainability

The evaluation found that communities with an active water and sanitation committee that provides equitable roles to all members, and an adequate financial plan for covering operation and maintenance costs were crucial components for the sustainability of improved water sources.  Qualitative data demonstrated that committee roles and membership were not balanced equitably according to gender and that the pricing plans were often inadequate to cover the costs of operations and maintenance.

 

Effect on household income attributable to MCC N/A

 

The evaluator conducted qualitative research to inform the final evaluation.  The supplementary qualitative study included 10 focus group discussions and 20 key informant interviews which totaled 116 participants, across 15 communities – 10 from the intervention group and 5 from the control group.

 

Qualitative
Evaluation Period All of the water points were completed by January 2012. The evaluation assumes that by 2014 the benefits of water activity might be visible and measurable in the communities.  Qualitative data collection took place in August 2014.
Outcomes Community Involvement

·         The siting of water points was based on technical and expert decisions on the most appropriate location; communities were not necessarily consulted.  The evaluation found that consultations that did take place were with community representatives (traditional authorities, unit committee members, etc.) rather than community-wide.

·         In spite of the low level of education among rural dwellers, they are not entirely naïve about technical issues, and desire to be involved in critical matters that affect them and their community.  The evaluation found that community members’ reservations about the project might have been alleviated if they had been involved in decisions on the source of the project, funding arrangements and terms, and management.

 

Direct and Indirect Impacts

·         In intervention communities, the evaluation found that the projects have largely resulted in the provision of clean water used for drinking and other domestic purposes. Communities, which previously depended on unimproved sources of drinking water, now consider pipe water and boreholes as their predominant sources of water.

·         As long as the new water infrastructure was functional, respondents reported that they obtained and, actually used, more water than before.

·         An unexpected effect of the program, as mentioned by some participants, might be a potential reduction of traffic accidents as result of people crossing fewer roads on a regular basis to collect water.

·         According to focus group respondents, many of the illnesses related to water points that were prevalent in the communities have been perceived to have been nearly eliminated from the communities studied.

 

Sustainability

·         With one exception, water committees in the remaining intervention communities were operational. The water committees in most communities had established rules to ensure cooperation, maintenance, and sustainability of the water infrastructure, including cost-reflective payment systems.  However, the cost of electric energy arose as an obstacle to the sustainability of the water points and committees.

 

Gender

·         Despite the fact that women play the primary role in fetching water, the evaluation found that they were underrepresented in consultations and as part of the management committees around the water points.

 

Effect on household income attributable to MCC N/A

 

Lessons Learned

  • Projects should be designed to address the root cause of a clear and evidence-based problem. In this activity, MCC assumed that changes in the time to gather water would result in greater productivity and increased school attendance.  One of the main benefit streams in the project was time savings, but it is unlikely that a reduction of 13 minutes per day in gathering water will amount to large changes in productive uses of time.  If the amount of time savings is small, we should think more critically about how we value time savings and what kinds of behavior changes are reasonable to expect as a result of time savings.  MCC has addressed this lesson by implementing new project design guidance which has sections that focus on problem analysis.
  • There was not a good understanding of the local context (community expectations or current water collection practices) during project design and implementation. Water points were placed for primarily hydro-geological considerations without community consultation or any effort to maximize time savings.
  • It is important to have a good sense of the counterfactual before the project starts and to monitor the counterfactual assumptions during implementation. One of the key predicted benefits of the project was reducing the incidence of Guinea worm. However, during implementation, Guinea worm was eradicated in Ghana.  This highlights the importance of updating assumptions around the counterfactual to ensure that the project results are accurate.
  • From an evaluation perspective, qualitative studies should be better focused. MCC requires evaluation questions for impact and performance evaluations, and qualitative studies should share these primary evaluation questions.  MCC has addressed this lesson by updating the evaluation design report template to specifically include guidance on qualitative evaluation reports.

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. This figure is an estimate based on financial obligations reported by the Millennium Development Authority (MiDA) in the internal Compact Completion Report (p. 148).
  • 2. These figures are based on MCC obligations as of February 2016.
  • 3. Indicators that have a completeness score above 95% are considered complete for the purposes of the completeness rate calculation.
  • 4. These figures are calculated using all non-evaluation indicators with targets in Ghana I Rural Development Project/Community Services Activity/Water and Sanitation Sub-Activity.