Measuring Results of the El Salvador Rural Electrification Sub-Activity

The MCC compact with El Salvador was a five-year investment (2007-2012) of $449.6 million. The $30.34 million Rural Electrification Sub-Activity is the subject of an independent impact evaluation summarized here. There was a small household solar panel component of the Rural Electrification Sub- Activity which will be the subject of a different evaluation.

  • The overall program logic of the Rural Electrification Sub-Activity was to improve access to the electricity network and shift use of energy alternatives to This shift was expected to decrease indoor air pollution and improve health outcomes which in turn, would lead to increases in households’ economic productivity and incomes.
  • The evaluation results showed that households that received a subsidy to cover a portion of the costs to connecting were more likely to get a formal connection to the grid and that connected households used electricity for lighting instead of As a result, indoor air pollution and the incidence of acute respiratory infection among children under 6 declined.
  • In addition, children in connected households were more likely to participate in education activities, such as studying at home, spending time at school, and going to and from school. Adults were more likely to engage in self-employment and in non-agricultural In particular, women increased their probability of operating a home business.
  • Finally, from the experimental analysis there is a strong indication of an increase in income for households that connected because they received a voucher; however, the point estimate is not statistically The non-experimental effects suggest an increase of 55 USD in non-labor net income (18 percent increase from baseline) and 208 USD on labor net income (20 percent from baseline). 5 The effect on total net income is 111 USD (8.8 percent of the baseline).
  • Through this evaluation, it was learned that in contexts similar to El Salvador where connection rates are high, lowering the up-front costs for formal household electricity connections can be effective at increasing such connections and behaviors around use of alternative energy sources can change
  • This evaluation is complete and there are no planned next The household solar panel component of the Sub-Activity is the subject of a different evaluation expected to be released in 2017.

In Context

The MCC compact with El Salvador was a five-year investment (2007-2012) of $449.6 million in three projects: the Connectivity Project, the Human Development Project and the Productive Development Project. The Human Development Project included two major activities, the Education and Training Activity and the Community Development Activity. The Community Development Activity consisted of three sub-activities: Rural Electrification, Community Infrastructure, and Water and Sanitation. The

$30.34 million Rural Electrification Sub-Activity is the subject of an independent impact evaluation released by MCC in September 2017, the results of which are summarized here. Under this Sub-Activity, there was also the distribution of off-grid household solar systems and technical assistance for community capacity building to ensure system maintenance and sustainability. The off-grid solar system is not included in this impact evaluation, but will be covered later in a performance evaluation. The whole Sub-Activity represents 7 percent of the total compact. Other components of the compact are the subject of already published and forthcoming independent evaluations.

Compact Spending: El Salvador Electricity

Program Logic

The Rural Electrification Sub-Activity was designed to address the cost of access to high quality electricity. Its objective was to improve the lives of Salvadoran living in the Northern Zone through the construction and extension of distribution lines and individual household connections to electrical networks. It was expected that improved access to the electricity network would lead to electricity use instead of sources of energy like wood and kerosene. This in turn would decrease indoor air pollution and improve health outcomes which would lead to increases in households’ productivity. In addition, access to electricity would allow household members to allocate more time to productive activities and open up non-farm business opportunities. These changes would imply income flows that are more diverse and perhaps less volatile, promoting resilience and helping households out of poverty.

Flow Chart: Program Logic

There were several key assumptions underlying the Rural Electrification program logic during the design of the investment:

  • Households without electricity spent money on candles, batteries for lamps, and kerosene to obtain alternative energy to light
  • Households would be able to pay the certification fee so that the household could be connected to the
  • Increased access to electricity will lead to reduction in alternative energy sources that are harmful to health, such as the use of kerosene lamps and the burning of
  • With access to electricity, households will watch TV, use the light to read, and participate in other leisure
  • Households would have resources to open new household businesses.
  • Children of households with access to electricity will have more hours of light to study and complete school

For the complete program logic of the Rural Electrification Sub-Activity, please refer to the final evaluation report.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators Level Baseline (2007) Actual Achieved (Compact End Date) Target Percent Complete
Households with electricity in the Northern Zone (percentage) Outcome 78 90.06 90 100.5%
Households benefited with the connection to the electricity network Output 0 35,412 24,426 145%

Source: January 2013 Closeout ITT which includes data through the end of the compact, based on reporting from FOMILENIO, electricity distributors, and the National Department of Statistics.

 

The average completion rate of output targets is 145 percent and targets were met or exceeded in all output indicators. The average completion rate of outcome targets is 100.5 percent and targets were met or exceeded in all outcome indicators.  1

Evaluation Questions

The evaluation was designed to answer the following questions:

  • What is the impact of electrification on the cost of energy and energy consumption?
  • What is the impact of introducing energy-efficient technology (i.e. connection to the grid vs other sources of off-grid energy) on uses of electricity?
  • What is the impact of electrification on time allocation?
  • What is the impact of electrification on indoor air quality?
  • What is the impact of electrification on productive activities?
  • What is the impact of expanded access to and use of electricity on household economic welfare?
  • What are the differential impacts for women men?
  • Are results likely to be sustainable?  2
  • Why do we see the impacts we see?

The evaluation covered the primary benefit streams that were modeled in the economic analysis of the program. More detail on this topic can be found in the Evaluation Design Report here: https://data.mcc.gov/evaluations/index.php/catalog/111U33T

 

Evaluation Results

The Rural Electrification Sub-Activity impact evaluation was based on two main empirical strategies to identify the effects of electrification on the outcomes of interest: (1) random encouragement and (2) difference-in-difference (fixed-effects estimation). For the first strategy, vouchers were provided to 400 randomly-selected households to help pay a portion of the amount needed for their internal wiring and connection fee for a safety certification that was needed before the household could receive an electricity connection (a cost not covered by the intervention). This approach both helped to ensure that households were able to take advantage of the electricity connection provided by the program and introduced randomization, which allowed for a more rigorous estimation of results that are attributable to the intervention. The second strategy compares households that receive electricity later (or that  never received it during the evaluation period) with households that are connected early on. The results from each of these methodologies are presented in one table below because the two methodologies produced similar results.

Evaluator Social Impact
Impact or Performance? Impact
Methodology Random Assignment and Difference-in-Difference
Evaluation Period Implementation: September 2009 to September 2012

Data Collection Rounds: 2009, 2010, 2011, 2012 and 2013. Exposure Period: From one to four years

Short-term Outcomes Connections

·         Voucher recipients were between 11 and 19 percentage points more likely to get a formal connection to the grid than control households.

·         A $10 reduction in the connection fee increased the probability of connection by two percentage points, though no systematic difference was found between those receiving the high and low discount vouchers, suggesting diminishing returns to increasing voucher discount.

·         Neighbors were more likely to connect if they saw nearby households connecting. An additional connection within 100 meters increased the probability of a single household having a connection by 10 percentage points, almost the same effect as the household itself receiving a voucher.

·         Households with an informal connection were significantly more likely to connect. Households with informal electricity at baseline were 18 percentage points more likely to take up the voucher, as were households with a property title (eight percentage points) and households with floor other than dirt (10 percentage points).

Cost, Consumption and Use

·         Decreased likelihood of using non-electric lighting sources, in particular kerosene for lighting. Households were 24 to 33 percentage points less likely to use kerosene or spend any money on kerosene.

·         No evidence of changes in cooking practices, either in the use of wood for cooking or in the probability of cooking outdoors.

 

Medium and Long-term Outcomes Time allocation

·         Electrification increased the probability of school-age children participating in education activities (studying, time at school, going to and from school) by 78 percentage points.

Indoor Air Quality

·         Overnight air pollutant concentration was 67 to 73 percent lower among voucher recipients, due to substitution away from kerosene as a lighting source.

·         Acute respiratory infections among children under the age of six reduced by 37-44 percent.

Objective-level Outcomes Productive Activities

·         Electrification led to increased ownership of refrigerators (54 percentage points), blenders (25 percentage points), and washers (13 percentage points).

·         Beneficiaries of electrification were more likely to engage in self-employment and in non-agricultural activities. Non- agricultural independent work in the four weeks leading up to the survey increased by 13 percent among voucher recipients.

·         Electrification increased the probability of operating a home business by 12 percentage points. The effect is concentrated among women, with women from on-grid households being 25 percentage points more likely to operate a home business.

Effect on household income attributable to MCC ·         The evaluation found mixed evidence on the effects of electrification on household income. From the experimental analysis there is a strong indication of an increase in income for households that connected because they received a voucher; however, the point estimate is not statistically significant. The non-experimental effects are more modest and more precisely estimated. The non-experimental effects suggest an increase of 55 USD in non-labor net income (18 percent increase from baseline) and 208 USD on labor net income (20 percent from baseline).  3 The effect on total net income is 111 USD (8.8 percent of the baseline). The differences across these estimates show that the effects could be very large for the households that connected to the grid because of the voucher.

 

Lessons Learned

Lesson #1: In contexts similar to El Salvador, lowering the up-front costs for formal household electricity connections can be effective at increasing such connections. The evaluation showed that in El Salvador where electrification rates were already close to 80% of the population in the Northern Zone, and where household income was around $2,400, providing a subsidy of 20% or 50% of the connection fee to a household for an electricity connection was effective at increasing connections. 4  Connection fee subsidies may be viable in other countries with similarly high rates of electricity access.

 

Lesson 2: In contexts similar to El Salvador, households in areas with numerous informal connections are more likely to establish formal connections when they see their neighbors formally connecting. This finding implies that funds could be used more efficiently by strategically connecting some households in areas with a large number of informal connections.

Lesson #3: In contexts similar to El Salvador, rural electrification can change behaviors rapidly. It can replace the use of alternative energy sources in a short period of time. The evaluation showed that households in El Salvador switched quickly from using kerosene for lighting to electricity and it had a significant impact on air quality and health. Similar changes could be expected in countries with high formal and informal electrification rates.

 

As a result of these lessons, MCC will do the following:

  • When developing projects to increase formal connections to the grid, MCC will consider (i) subsidizing the connection fees of the consumers but in a way that does not exacerbate financial health of the power utility that incurs the cost of connections and (ii) targeting, first, those areas with greater informal connections who may opt to switch to formal connections via the spillover effects observed in El Salvador.
  • MCC will consider working to measure and quantify income benefits – whether from health improvements, educational improvements or job creation — resulting from the consumer’s switching of fuel source for some of his or her activities to electricity and consider, as the context warrants, the inclusion of some of the benefits found in El Salvador in the economic analysis.
  • Work together to define research questions for energy evaluations up-front and organize implementation and the evaluation so that those questions can be This could include testing different approaches for increasing connections like different subsidy levels or providing service drops to households in some areas and not in others (assuming funds wouldn’t be sufficient to provide them everywhere).
  • Ensure that there are standardized modules for measuring energy consumption in future energy evaluations to allow for comparison of results across countries.

Next Steps

This evaluation is complete and there are no planned next steps. The household solar panel component of the Sub-Activity is the subject of a different evaluation expected to be released in 2017.

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets in the Rural Electricity Sub-Activity.
  • 2. Note that the evaluation did not end up answering this question explicitly.
  • 3. Not every household has both labor and non-labor income, so those two estimates of increased income cannot be added together.
  • 4. There was no statistically significant difference in the connection rates between the households receiving the 50% subsidy and the 20% subsidy.
  • 5. Not every household has both labor and non-labor income, so those two estimates of increased income cannot be added together.