Program Overview
MCC’s $177.7 million Armenia Compact (2006-2011) funded the $8.4 million Rural Road Rehabilitation Project to improve the quality of 24 kilometers of rural roads. The project was designed on the theory that improved rural roads would reduce transportation costs, enhance residents’ access to markets and social infrastructure, and increase vehicular activity, thereby increasing employment opportunities (both short- and long-term) and agricultural investment and production, eventually resulting in increased household income. In addition, about a quarter of the project cost went to fund the design of 430 kilometers of rural roads that were subsequently rehabilitated by the World Bank and the Government of Armenia.
Evaluator Description
MCC commissioned Mathematica to conduct an independent final impact evaluation of MCC’s Rural Road Rehabilitation Project which contributed early designs to the subsequent road works funded by the World Bank and the Government of Armenia. Full report results and learning: https://data.mcc.gov/evaluations/index.php/catalog/100.
Key Findings
Road Quality
- Households in communities served by the rehabilitated roads perceived improved road quality relative to the comparison group.
Access to Markets and Social Infrastructure
- The rehabilitated roads improved market access relative to the comparison group; however, social infrastructure, such as hospitals and schools, was no more accessible than for the comparison group.
Construction Employment
- The road rehabilitation had no detectable short-term employment income impacts for nearby households relative to the comparison group.
Agricultural Production and Sales
- Households near the rehabilitated roads were more likely to use roads to buy agricultural inputs but spent less time on them selling their products.
- No impact was found on agricultural production or sales relative to the comparison group.
Income and Poverty
- The evaluation found no impact on income relative to the comparison group, and an unexplained increase in poverty among the households near the roads relative to the comparison group.
Evaluation Questions
This final impact evaluation was designed to answer the following questions:
- 1
Did rehabilitating roads affect the quality of roads? - 2
Did rehabilitating roads improve access to markets and social infrastructure? - 3
Did rehabilitating roads improve income from employment? - 4
Did rehabilitating roads affect agricultural productivity and profits, and if so, by how much? - 5
Did rehabilitating roads improve household well-being for communities served by these roads, especially income and poverty?
Detailed Findings
Road Quality
The evaluation found treatment households were 39 percentage points more likely to rate the quality of regional roads as good or excellent than comparison households did.
Access to Markets and Social Infrastructure
Treatment households were 20 percentage points less likely than the comparisons to report problems with market access. Having better roads also increased their usage for shopping and family visits by 17 percentage points.
Construction Employment
The World Bank estimated that their project generated 10,000 person-months of employment during the time period covered by the evaluation. Though the evaluation did not detect an increase in short-term employment incomes.
Agricultural Production and Sales
Although households near the rehabilitated roads were no more likely to use roads and spent less time on the roads selling agricultural products, they were nine percentage points more likely than comparison households to use roads to buy agricultural inputs. Despite these intermediate effects, the improved roads had no detectable impact on agricultural production or sales.
Income and Poverty
The improved roads did not have an observable impact on income within the short evaluation time frame, but curiously poverty increased 12 percentage points among male-headed households; a result that remains unexplained.
MCC Learning
It is important to consider alternative interventions that may prove to be relatively more cost effective and economically viable than simply paving a road to achieve agricultural outcomes.
Project teams must ensure complete and high-quality data is collected both for cost-benefit modeling that feeds into project selection and design, and for M&E purposes during and after implementation.
Base investment and evaluation decisions on a clear program logic.
Set realistic time horizons and keep data collection plans flexible.
Ensure sufficient statistical power in order to measure realistic changes in key outcomes.
Evaluation Methods
This final impact evaluation used the difference-in-difference methodology. The treatment group comprised communities served by the 27 road links that were in MCC’s original plans and were rehabilitated in 2009 and 2010 with World Bank and Government of Armenia funding. The comparison group comprised communities served by the 28 road links that were in MCC’s plans, but were not rehabilitated as of 2011.
The difference in outcomes for the rehabilitated roads (treatment group) before and after rehabilitation was compared to the before and after difference in the non-rehabilitated roads (comparison group). The impact of the road rehabilitation was assessed one to two years after it was completed.
Data from the Integrated Living Conditions Survey conducted by the National Statistical Service of Armenia was used for the evaluation. This survey sampled a cross-section of 4,848 households from the treatment and comparison road communities annually from 2007 to 2011.
2020-002-2446