Summary: Measuring Results of the Mongolia Energy and Environment Project Stove Subsidies Component

In Context

The MCC compact with Mongolia was a five-year investment (2008-2013) of $285 million in five projects: (i) the Health Project, (ii) the Property Rights Project, and (iii) the Vocational Education Project, (iv) the North-South Road Project, and (v) the Energy and Environment Project.

On April 27, 2009, less than 1 year after the compact entered into force, the Government of Mongolia notified MCC that it intended to withdraw one of the original projects, a Rail Project, from the Compact. In January 2010, approximately $188 million from the Rail Project was reallocated for the expansion of the Health, Property Rights, and Vocational Education Projects, and the addition of the new North-South Road and Energy and Environment Projects, leaving approximately 3.5 years for their implementation.

The Energy and Environment Project (EEP) included three major activities: the Millennium Challenge Energy Efficient Innovation Facility (MCEEIF) ($33.8M); Wind Activity ($3.0M); and a Public Awareness Campaign ($2.0M). The $33.8M MCEEIF Activity accounted for 13% of the total Compact investment, and 84% of total EEP disbursements. The focus of this Summary of Findings is the stove subsidy component of the MCEEIF Activity. Results are based on the findings of an independent impact evaluation contract signed between the MCC Department of Policy and Evaluation’s Monitoring and Evaluation Division and Social Impact; a consulting firm specialized in conducting performance and impact evaluation of development programs. The results of the impact evaluation are available in the final evaluation report titled “Impact Evaluation Results of the MCA Mongolia Energy and Environment Project Energy-Efficient Stove Subsidy Program”, dated August 20, 2014.

Program Logic

Ulaanbaatar, Mongolia has been referred to as both the coldest capital city and the second most polluted city in the world. Air quality is worst in the winter, when average temperatures range from -40F (-200C) during the day to -400F (-400C) at night. Average daily concentrations of fine particulate matter (PM2.5) 1 can be 15 times higher than the World Health Organization’s guidelines established to minimize morbidity and mortality risk. During the winter months (October-March), residents of Ulaanbaatar’s peri-urban “ger district” 2 typically use coal-burning stoves to heat their traditional homes (gers) and houses on a nearly continuous basis. The heavy use of coal in residential stoves is a major source of air pollution that is estimated to contribute up to 70% of PM2.5 in the ger district (World Bank, 2009). Visibly poor air quality concerns Ulaanbaatar residents, the Mongolian Government, and global air quality experts. The poor, under-nourished, very young, very old, and individuals with pre-existing respiratory disease and other ill health related aliments are especially at risk. It is also noted that the economic burden of stove fueling is a significant barrier to poverty reduction as annual fueling expenses can amount to 40% of the average household income of the poorest income quintile; an average household consumes approximately 4.2 tons of raw coal and 4.7 cubic meters of wood during a single heating season.

In partnership with MCC, MCA-Mongolia introduced the stove subsidy in 2011 to help reduce air pollution through financial support for the promotion and use of more energy efficient stoves. The subsidy was intended to incentivize purchases of efficient stoves demonstrated to reduce stove particulate matter emissions, thus improving health outcomes. In addition, it was proposed that increased stove efficiency would reduce fuel use and consequently fuel expenditures. Collectively, these effects were expected to contribute to economic growth and poverty reduction.

While the Millennium Challenge Energy Efficiency Innovation Facility also included funding for heat only boiler (HOB) replacement, subsidies for insulation, subsidies for energy efficient homes, and small grants for greening, air quality research, and the Energy and Environment Project also included a Wind Activity and Public Awareness Activity, these were not analyzed as part of this evaluation.

 

Key assumptions underlying the EEP theory of change for stoves:

  • On-going stove development activities of other donors will result in a technically and commercially viable stove and/or a technically and commercially viable stove can be found and marketed in Mongolia.
  • A “one size fits all” implementation approach did not work in the past; offering a variety of solutions to consumers and an intense public awareness and informational campaign will maximize participation.
  • Decreased fuel expenditures are dependent on type of fuel used, stove usage behavior, and insulation
  • Demand is price elastic, that is, quantity demanded is very responsive to lower prices.
  • All eligible ger district residents, both existing and new, will be eligible to receive subsidies.
  • Purchasers of stoves will be willing to sign a Consumer Participation Agreement requiring the trade-in for disposal of old stoves.
  • New stoves will lead to decreases in emissions and improvements in ambient air quality sufficient to impact health.
  • Nalaikh coal will continue to account for the majority of household fuel. Alternate fuels (semi-coked coal) will not be used in subsidized stoves if not approved for use by the manufacturer.
  • Men and women are equal participants in the stove market.

 

Measuring Results

MCC uses both monitoring and evaluation data for measuring and reporting results. Monitoring data is typically generated by the program implementers and specifically tracked households that purchased MCA subsidized stoves. However, monitoring data is limited in that it cannot tell us what program participants would have done in the absence of the MCC-funded intervention. Accordingly, MCC invested in an independent impact evaluation to establish a counterfactual in order to assess what would have happened in the absence of the MCC investment.

The following table summarizes performance on output and outcome indicators specific to the stove activity according to the Compact monitoring data. 3

Monitoring Indicators Tracked During Implementation of the Rural Business Development Project

Indicators Level Baseline Actual Achieved Target Percent Complete
Percent difference in PM2.5 emissions Outcome 0 -28% -57% 49%
Absolute difference in total ambient PM2.5 contributions from MCA stoves versus traditional stoves in gers and standard houses (tons) Outcome 0 -1,150 -2,635 44%
Percent difference in average nighttime household PM2.5 emissions for households with MCA versus traditional stoves 4 Outcome 0 -65% -86% 76%
Percent reduction in median fuel costs 5 Outcome 0 -7% -15% 47%
Carbon monoxide concentration in project households (ppm) 5 6 7 Outcome 4.5 ppm 3.6 ppm 4.5 ppm 120%
PM2.5 concentration in project homes (milligrams per cubic meter) 8 9 Outcome 0.16mg /m3 0.16mg /m3 0.16mg /m3 100%
Percent difference in raw coal consumption 5
Number of consumers purchasing subsidized products Output 0 128,681    
Number of female consumers purchasing subsidized products Output 0 50,084    
Number of subsidized stoves sold Output 0 103,255    
Subsidized stoves in participating homes (%) 10 Output 0 81 90 90%
Total subsidies provided (USD millions) Output 0 29.65    
 

 

Of the seven outcome indicators, seven had established targets, two achieved their targets. Of the five output targets, only one had an established target, which was not achieved.

 

The output indicator related to females purchasing stoves was important for monitoring the extent to which both males and females were able to actively benefit from the program, which was an assumption underlying the program logic. Female participation was relatively strong in the program, and a high share of female participants were from female-headed households. Female-headed households made up 29% of total beneficiaries receiving subsidies (for a stove or other product).

The average completion rate of output and outcome targets is 67%. This figure does not take into account the four indicators without targets. Targets were achieved or exceeded in 2 of the 12 combined output and outcome indicators; equivalent to 17%.

Evaluation Questions

The evaluation was designed to answer two primary questions, each with its own set of sub-questions:

  • How do energy-efficient products impact ambient air pollution levels, as well as the health and income of residents in Ulaanbaatar?
    • How does the use of MCA stoves affect fuel usage and expenditures?
    • Does the use of MCA stoves affect available household income?
    • What is the impact of MCA stoves on emissions of CO and PM2.5?
    • What would be the estimated change in health for Ulaanbaatar residents?
    • How do MCA stoves affect household expenditures related to respiratory health problems?
  • How do different MCA stove models and different patterns of usage affect the level of impact on ambient air pollution, health and income of households with MCA stoves?
    • Do different MCA stove model types impact fuel expenditures, income, and PM2.5 emissions under typical usage behavior?
    • Do deviations from expected MCA stove usage patterns impact air pollution, health, and income of households with MCA stoves?
    • Did the MCA stove program result in differential impacts on men and women?
    • Does possession of additional energy efficient products such as vestibules or additional ger insulation modify the impact of MCA stoves on ambient air pollution, health, and income?

 

Evaluation Results

As a market-based intervention, households chose whether or not to purchase an MCA stove (or other subsidized product). Since a randomized intervention assignment was not possible and the evaluation was implemented after the project had started, a quasi-experimental propensity score matching (PSM) design was selected to adjust for differences between those who did or did not choose to purchase an MCA stove. Matching on propensity scores enabled construction of treatment and comparison groups that were balanced along observed household characteristics, thereby providing a counterfactual for the intervention.

While key results are presented in the table below and available in the report, it is important to note to the findings: First, evaluation findings note that households using MCA stoves experienced little to no fuel savings but that they enjoyed statistically significant 11 higher average nightly indoor temperatures, suggesting that users may be sacrificing fuel economy for comfort. Overnight indoor temperature measurements (when most household members are likely to be home and the dwelling requires consistent heating) during the latter (colder) part of winter showed MCA stove owners maintained 17.55°C (63.59°F) indoor temperatures compared to traditional stove owners, who had 15.70°C (60.26°F).

Second, when examining the gender dynamics of stove preference and usage of the sampled population, the data indicate that the impacts of this activity did not systematically vary by gender, confirming the assumption underlying the program logic. No significant gender differences between males and females were observed for a) coal consumption, b) type of coal used, c) compliance with operation instructions, d) opinions of MCA stove performance, or e) those experiencing illnesses. However, it is noted that amongst female-headed households only, MCA stove owners were found to spend (on average) 20.6% more on coal every month when compared to traditional stove owners. Project results indicated that female clients and female-headed households were more likely to take advantage of bank loans (beneficiaries could elect to make a purchase via cash or via loan). While female-headed households comprised 29% of sales, 38% of these women selected to purchase stoves using loans. Impact evaluation data indicate that female-headed households were more interested in the advertised fuel savings and were twice as likely to cite the subsidy and the resulting low price as a reason to purchase the stove, perhaps due to being in a financially more vulnerable position. This suggests that the subsidy itself may have been a crucial component in ensuring benefits to females and female-headed households. Among those who did not want to purchase an MCA stove, a larger proportion of females cited difficulty with cooking as a primary reason.

 
Evaluator Social Impact
Evaluation Type Impact
Methodology Quasi-experimental propensity score matching
Exposure Period Data were collected 3 times from a panel of households across the 6 winter heating months of October 2012 to March 2013. 12
Fuel consumption No significant difference was observed in fuel consumption between MCA stove users and traditional stove users. 13
Fuel expenditures No significant difference was observed in fuel expenditures between MCA stove users and traditional stove users 14
Health Health impacts were not directly measured; 15 however, using known dose-response functions (the most advanced methods to estimate health impacts currently available), it is estimated that PM2.5 emissions reductions measured through this study resulted in reductions in the incidence of various respiratory-related diseases including cancer (-9%), COPD (-8%), ALRI in children 0-4 years (-8%), ischemic heart disease (-5%), and stroke (-2%). This would have resulted in 47 avoided deaths and 1,643 disability-adjusted life years (DALYs).

Lessons Learned

There were several key lessons learned from this evaluation for MCC and future partner countries to consider when designing and implementing stove subsidy projects and evaluations.

  • Investing in stove subsidies to replace traditional coal-burning stoves with more energy efficient coal-burning stoves is a cost-effective intervention. The impact evaluation results demonstrate reductions in emissions that exceed the assumptions made in MCC’s ex-ante economic rate of return analysis. This suggests the EEP project design, at least for stoves, to be a cost- effective air pollution abatement measure.
  • More comprehensive pilot testing is required outside a laboratory setting before implementing a similar project at scale. The relationship between fuel efficiency and stove operation is direct and complex. Fuel efficiency gains observed in the laboratory and field testing prior to implementation were achieved under normal operating conditions. The evaluation observed more varied operating procedures among MCA stove owners. The result was observation of little to no fuel savings, warmer temperatures in the home, and/or fuel savings that were less than those achieved in the laboratory (30%). While demonstration and written instructions were provided to consumers, alternate methods or greater investment in the same is recommended. Laboratory level fuel savings are still attainable by stove owners who operate their stoves according to the instructions.
  • Need to assess potential user preferences in order to inform marketing and communications. While the project undertook both laboratory and field testing to select stoves for inclusion in the subsidy program, the difference in results between ex-ante findings and evaluation findings concerning fuel savings suggests the need for a more thorough ex-ante assessment of consumer preferences and variation in actual stove operation, particularly differences between men and women, to better guide product demonstration and training activities.
  • Need to consider how changes in the fuel supply market may affect project design. Stove performance was tested with the predominant coal available in the target market which was in turn prescribed for use with MCA stoves. The evaluation showed that other coal fuel was also used in the target market albeit to a significantly lesser extent. In the last year of the program, the mining operation which provided the predominant coal type was closed and an alternate coal type was introduced by the Government with subsidy. Had more time been available, the project should have developed tests and strategies for alternate stove/fuel combinations.
  • Need to consider how bundling strategies may incentivize behavior change. Given the impact of insulation, operations, and fuel type on stove performance, the project may have benefited from additional testing and made better use of “bundling” strategies to incentivize the purchase of energy efficiency packages, rather than individual products. Meaningful bundles might have combined the purchase of stoves with insulation, cooking devices, fuel, cooking pots, etc.
  • More coordination on marketing and communications may be required to mitigate political risks. A successful project outcome in the form of air pollution reduction was dependent upon maximizing product sales. During the election period, incorrect and negative statements were made about the project in the press. Moreover, traditional stove producers and alternative fuel producers who were perhaps threatened by the introduction and subsidy of new coal-fired stoves, criticized the program. This misinformation may have created some confusion in the market about the efficacy of the program and its benefits. While stove sales remained high overall, the project would have benefited from improved implementation of, and coordination with, the Government’s national plan to reduce air pollution and its communication strategy.
  • Ensure monitoring indicators are consistent with project design. The design of this project as a funding facility might have been more conducive to the inclusion of a greater number of process indicators early on, and the addition of relevant output and outcome indicators later on. Similarly, with respect to emissions and air pollution reduction, targets should have been set after testing and modeling revealed what was achievable.

 

Next Steps

As part of follow-up evaluation closeout activities, the independent evaluation team is planning to conduct a workshop in Mongolia in order to more broadly disseminate the research findings among relevant stakeholders in early 2015.

Footnotes
  • 1. PM 2.5 is particulate matter smaller than 2.5 micrometers. Generally emitted through combustion, particulate matter is associated with reduced lung function, asthma, coughing, difficulty breathing, and cardiovascular events like heart attacks and chronic obstructive pulmonary disorder.
  • 2. Many migrants to the capital city have established their gers in outskirt areas of Ulaanbaatar, thereby receiving the name “ger district.” However, both gers and standard houses are located in this area.
  • 3. These tables reflect data from the final approved closeout MCA-Mongolia Indicator Tracking Table (ITT). Please note the Evaluation Report references data from earlier ITTs which may explain some minor discrepancies in results statements.
  • 4. This indicator represents emissions measured from approximately 5pm through 9am, weighted by the number of each stove type sold in Ulaanbaatar.
  • 5. This result is not statistically significant, indicating that no meaningful difference between traditional and MCA stoves could be detected by this evaluation.
  • 6. Original target specified as “Equal to traditional stoves.” This target is set equal to the baseline to capture the intention that levels of CO concentration in households (HHs) with MCA stoves not exceed that of HHs with traditional stoves.
  • 7. The result is expressed textually as it is not mathematically possible to express in numerical terms as the end of compact target is equal to the baseline value.
  • 8. Original target specified as “Equal to traditional stoves.” This target is set equal to the baseline to capture the intention that levels of PM2.5 concentration in HHs with MCA stoves not exceed that of HHs with traditional stoves.
  • 9. The result is expressed textually as it is not mathematically possible to express in numerical terms as the end of compact target is equal to the baseline value.
  • 10. Proportion of participating households (defined as having an MCA stove in the home) that actually utilize the MCA stove.
  • 11. The difference of 1.84°C (3.33°F) is statistically significant at p=0.029
  • 12. Preliminary data were collected in the second half of the winter of 2011-2012 in order to calibrate measurement methods and sample sizes.
  • 13. MCA stove users performed 0.33 fewer daily fueling events than traditional stove users (p<0.001) but utilized 0.72 kg more coal (p=0.001) on average at each fueling, a 14% increase, thereby equalizing total daily quantities of fuel used by the two groups. However, when fuel consumption was compared between MCA stove users correctly following use instructions and traditional stove users, MCA stove users were observed to have 17% reductions in daily coal consumption (p<0.01).
  • 14. MCA stove owners as a whole spent MNT 1,055 less on coal each month than traditional stove owners, or an estimated 6,333 less across six months of winter, but this difference was not significant.
  • 15. The survey methods and the sample size were not designed to measure health outcomes. Respondents were asked to report current respiratory, cardiac, and dermal symptoms experienced by household members in vulnerable age groups (<5 and >60 years old), to collect some data on symptoms that could be associated with air pollution. However, by design, no precise conclusions with regard to health could be made from these data, as these data represented self-reported symptoms rather than disease incidence.