Measuring Results of the El Salvador I Rural Electrification Sub-Activity Solar Panel Component

The MCC compact with El Salvador was a five-year investment (2007-2012) of $450 million. The Solar Panel Component of the $30 million Rural Electrification Sub-Activity is the subject of an independent performance evaluation summarized here.

  • Through the installation of 1,950 solar panel systems, the Solar Panel Component was intended to improve access to electricity, thereby reducing reliance on traditional sources of This in turn would decrease indoor air pollution, increase leisure time, and improve performance of children at school, which ultimately would lead to improved health outcomes and increased household incomes.
  • The evaluation found that of the interviewed participants, 78 percent of the solar panels were still in Participants reported using less kerosene, wood, and candles. All but one of the interviewed participants reported saving money on energy since they began using their solar panel systems.
  • Participants reported that the solar panel systems allowed for more time for activities primarily related to leisure, housework, and education. Women reported spending the additional time on housework (e.g. cooking and cleaning), while men reported using the additional time for leisure Nearly half of the participants in the sample explained that school age children have more time to study at night.
  • Nearly 90 percent of participants interviewed reported that air quality in their homes had improved greatly since they began using the solar panel systems; however, effects on health outcomes could not be determined with any degree of Finally, due to the limited electricity capacity, very few households reported using the extra time for income generating activities.
  • Through this evaluation, it was learned that household electricity demand should be assessed up- front in order to design the project to offer appropriate technology and that it is important to work closely with local organizations to ensure capacity for post project monitoring and follow-up.
  • This evaluation is complete and there are no planned next

In Context

The MCC compact with El Salvador was a five-year investment (2007-2012) of $450 million in three projects: the Connectivity Project, the Human Development Project and the Productive Development Project. The Human Development Project included two major activities, the Education and Training Activity and the Community Development Activity. The Community Development Activity consisted of three sub-activities: Rural Electrification, Community Infrastructure, and Water and Sanitation. The almost $2 million Solar Panel Component of the $30 million Rural Electrification Sub-Activity is the subject of an independent performance evaluation released by MCC in October 2017, the results of which are summarized here. The entire Rural Electrification Sub-Activity represents 7 percent of the total compact and the Solar Panel Component represents approximately 0.4 percent of the total compact. The on-grid portion of the Rural Electrification Sub-Activity is the subject of a soon-to-be released impact evaluation. Other components of the compact are the subject of already published independent evaluations.

Pie Chart: El Salvador Project Spending

 

 

*These figures are based on MCC obligations as of May 2017.

Program Logic

The Solar Panel Component of the Rural Electrification Sub-Activity was intended to improve the lives of those in remote communities in El Salvador’s Northern Zone that lacked access to the electrical grid network. Through the installation of 1,950 solar panel systems, recipients would have access to essential power that would allow them to run three lightbulbs, a cell phone charger, and/or small power appliances. It was expected that improved access to electricity would lead to reduced reliance on traditional sources of lighting, such as wood and kerosene. This in turn would decrease indoor air pollution, increase leisure time, and improve performance of children at school. Ultimately, this would lead to improved health outcomes and would increase household incomes through savings from cost of candles, kerosene and batteries.

Flow Chart: Program Logic

There were several key assumptions underlying the Rural Electrification program logic during the design of the investment:

 

  • Households will save the money necessary for maintenance of the solar panel
  • Increased access to energy will lead to a reduction in alternative energy sources that are harmful to health, such as the use of kerosene lamps and the burning of
  • Children of households with access to electricity will have more hours of light to study and complete school
  • With access to electricity, households will watch TV, use the light to read, and participate in other leisure

 

For the complete program logic of the Solar Panel Component of the Rural Electrification Sub-Activity, please refer to Annex I of the final evaluation report.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

 

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

 

 

Indicators

Level Baseline (2007) Actual Achieved (09/2012)  

Target

Percent Complete
Households benefited with the installation of isolated solar systems  

Output

 

0

 

1,950

 

1,950

 

100%

Source: (e.g. Closeout ITT from January 2013, which includes data through the end of the compact, based on reporting from FOMILENIO contractors.)

 

The average completion rate of output targets is 100 percent and targets were met or exceeded in 1 of the 1 output indicators.  1

Evaluation Questions

The evaluation was designed to answer the following questions:

  • To what extent was the Solar Panel Component of the RE Sub-Activity implemented according to the original plans?
    • Were the process and output targets achieved according to original plans and according to revised plans? If not, why?
  • To what extent did the Solar Panel Component reach the intended participants?
    • Were there any unintended participants?
    • Were some targeted households more likely than others to actually become participants?
  • What were the facilitators of implementation and what challenges were encountered?
  • What are the perceived or observed outcomes of the Solar Panel Component on the following:
    • Are households using the solar panels?
    • How have energy consumption patterns changed (i.e. sources of energy)?
    • Has the cost of energy changed?
    • How has total energy consumption by source changed?
    • Has household member time allocation changed (for women, men, children)?
    • Has air quality improved?
    • Has the household changed or increased their productive activities/income-generating activities?
  • What, if any, are the differential outcomes for women men?
  • How sustainable are the outcomes likely to be from the Solar Panel Component?
    • Are systems being maintained properly? Why or why not?
    • What types of households are able to maintain the solar panels?
    • Do households expect the grid to reach their homes in the near future?
  • What lessons can be learned from the solar component?
  • To what extent have the assumptions underlying the ERR held true?

 

The evaluation did not cover the following benefit stream that was modeled in the economic analysis of the program. More detail on this topic can be found in the Evaluation Design Report here:

33TUhttps://data.mcc.gov/evaluations/index.php/catalog/189U33T

  • Consumer surplus

 

Evaluation Results

The performance evaluation of the Solar Panel Component employed a non-experimental design that used multiple qualitative methods to collect and analyze ex-post data: document review, key informant reviews, direct observation, and in-depth interviews with 80 participant households.

Evaluator Social Impact
Impact or Performance? Performance
Methodology Ex-post Qualitative
Evaluation Period Program implementation: 2008-2010 Data collection: March-April 2017 Exposure period: 6-8 years
Outcomes Use of Panels

·         78 percent of interviewed participants still use their solar panels. Energy Consumption and Cost

·         Participants reported using less kerosene, wood, and candles. In fact, 50% of interviewed participants reported that they completely discontinued use of original lighting sources.

·         All but one of the 80 participants interviewed reported saving money on energy since they began using their solar panel systems.

Time Use

·         Participants reported that the solar panel systems allowed for more time for activities primarily related to leisure, housework, and education.

o  Women reported spending the additional time on housework (e.g. cooking and cleaning), while men reported using the additional time for leisure activities.

·         Nearly half of the participants in the sample explained that school age children have more time to study at night.

Air Quality

·         Nearly 90 percent of participants interviewed reported that air quality in their homes had improved greatly since they began using the solar panel systems.

Objective-level Outcomes Health

·         Although households reported improved air quality, effects on health outcomes could not be determined with any degree of certainty due to the nature of this evaluation.

Productive Activities

·         Fewer than 30 percent of the participants said that they use the solar panels for income generation. Due to their limited energy capacity, solar panels have scarcely stimulated productive work.

 

Lessons Learned

Lesson #1: Assess household electricity demand up-front in order to design the project to offer appropriate technology for different types of households. Conducting this analysis up-front would help determine households’ real energy needs to inform both the selection of solar panel capacity and participant targeting.

 

Lesson #2: Work closely with local organizations to ensure capacity for post project monitoring and follow-up. MCC should identify, and in some cases provide training for, local entities that can assume responsibility for following up and monitoring proper use of solar panels as well as facilitate access for participants to technical maintenance and spare components.

 

Lesson #3: Ensure that solar panels can be properly maintained by recipient households. MCC should ensure that two conditions are met: (1) that spare parts, batteries, etc. are available to households in their country at a reasonable cost and (2) that either households have sufficient resources to maintain the panels or the government is prepared to provide batteries or whatever may be required to keep the panels functioning properly.

 

Lesson #4: Document implementation in a detailed manner. At a minimum, MCC records should include logic models, any research that supported the program design, descriptions of the programs’ processes, significant changes and accompanying justifications, and basic budget information.

 

As a result of these lessons, MCC will do the following:

  • If MCC directly implements a similar project, MCC will consider (i) assessing the potential demand for electricity at a more granular level, provided MCC’s compact development process then applicable permits the cost and time required for such assessment and (ii) providing additional capacity building to the relevant partner organization for more sustainable operation & maintenance of the solar panels by the recipients.

 

  • If MCC indirectly implements a similar project (e.g., via providing a grant to an organization that in turn will supply solar panels), MCC will consider, in selecting the grantee, (i) the granularity of the assessment of the potential demand for electricity and (ii) the grantee’s plans for providing capacity building to the relevant partner organization for more sustainable operation & maintenance of the solar panels by the recipients.

 

  • MCC will develop a research agenda for solar project evaluations to fill gaps in evidence on the impacts of solar photovoltaic This research agenda will be considered during the design of future MCC solar projects. Part of this research agenda will include standard metrics for solar projects, which will allow for comparison of results across projects.

 

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets related to the Solar Panel Component under the Rural Electrification Sub-Activity.