Rehabilitating the Samtskhe-Javakheti Road in Georgia

Program Overview

MCC’s $395.3 million Georgia Compact (2006-2011) funded the $203.5 million Samtskhe-Javakheti (SJ) Road Rehabilitation Activity to address the SJ region’s economic isolation from the capital city of Tbilisi and the rest of the country. The activity aimed to reduce the time and costs associated with transportation by rehabilitating and constructing 220 kilometers of road. The improved road was expected to increase traffic volume, reduce travel times, and lower prices of consumer goods, ultimately leading to increased household consumption and higher levels of business investment.

Key Findings

Evaluation Questions

This final impact evaluation was designed to answer the following top-line question:

1. 1
   How does the road rehabilitation affect/cause economic development, new businesses, and economic and social integration in the region?

In addition, the following categories of outcomes were explored by the evaluation:

• Transportation related outcomes: traffic counts, vehicle speeds, travel times, and availability of public transport
• Investment, land use, and employment: industrial investment, land uses, cropping patterns, and employment
• Market prices: the prices of basic commodities on the local market
• Household welfare: income, consumption, and asset ownership
• Access to health and education: utilization of health care and education services.

Detailed Findings

Reader’s note: In interpreting these findings, it is important to bear in mind some limitations of the analysis. First, while the methodology corrects for selection bias to the greatest extent possible, the results could be driven by differences between the treatment and comparison groups. Additionally, the timeframe over which outcomes were measured was less than two years after the conclusion of the project. Thus, it is possible the results were unable to capture the full impact of the project on outcomes.

Travel and Transport

After construction was completed, the volume of traffic on project roads increased by an average of 44.2 vehicles per day (4.2 percent) relative to comparison roads, while the average speed along the roads increased by 13.6 km/h (24.4 percent). Additionally, there were large and significantly reduced self-reported travel times both to Tbilisi and to local markets where farmers sell their products. The difference-in-difference estimate indicated a reduced driving time to Tbilisi of 39.6 minutes and a 43.6 minute reduced driving time to local markets, while the continuous treatment model finds that a community located 30 minutes from the treatment road would experience a 83.6 minute reduction in travel time to Tbilisi and a 25.9 minute reduction to the local market.

Industrial Investment, Land Use, and Employment

At the community level, the improved roads in the region led to an estimated 26.9 percent increase in the number of industrial facilities (i.e. canneries, factories, agricultural processing facilities, and similar enterprises) in settlements near the project roads, although some of the impact may be due to selection bias. There were no observable changes in employment outcomes or in agricultural outcomes, such as cropping patterns or land use.

Market Prices

The evaluation found a complex relationship between prices and transport costs. In many cases, statistically significant findings were observed, but with different tendencies for different products, potentially related to depending on whether or not the product in question was produced in the local area. The strongest findings were for honey and beef. The project tended to cause honey prices to increase in areas where honey was produced and decrease prices in more distant markets. The opposite tendency was observed for beef; improving the road tended to lead to a fall in prices where beef was produced and a rise in prices of beef elsewhere.

Household Welfare

There was no evidence of impacts on household-level outcomes in the SJ region, such as income, consumption, or utilization of health and education services. One possible explanation is that the improved SJ roads did not impact these outcomes. However, in light of the significant findings observed at the community level, a more likely explanation is that impacts were not detected due to limitations of the household dataset, which was not ideally structured for the evaluation. Furthermore, insufficient time may have passed between the concluded construction and data collection for these outcomes to have begun to materialize.

MCC Learning

• 
  Base evaluation decisions on a clear program logic. The decision of when and what data should be collected should be driven by a clear program logic that underlies the investment decision.
• 
  Set realistic time horizons and keep data collection plans flexible. From the beginning, implementers and evaluators should build actions to mitigate risks associated with project delays into the evaluation design.
• 
  Ensure sufficient statistical power to measure realistic changes in key outcomes.
• 
  Include an upfront national or area-wide road network analysis based on selected criteria such as traffic volume, roughness index, and other parameters in the roads project selection process, as noted in the Principles into Practice: Lessons from MCC’s Investment in Roads.
• 
  Comprehensively address policy and institutional constraints to road maintenance and seek assurances from the partner government that the necessary mechanisms to ensure sustainability are in place prior to MCC’s investment.

Evaluation Methods

The evaluation used three methodologies to rigorously evaluate the causal impact of the program on outcomes. The first is a difference-in-difference methodology, whereby the project roads were matched to a set of similar comparison roads where no intervention had taken place. The difference-in-difference analysis compared traffic counts and socioeconomic outcomes for residents of communities located near the project roads and residents of communities located near the comparison roads. Secondly, the evaluation incorporated a continuous treatment approach to model project impact in a dose-response framework, so that communities located near project roads were assumed to experience greater impacts than those more distant. Finally, the evaluation estimated a matched difference-in-difference model, using propensity score matching to improve the comparability between treatment and comparison groups. The exposure period varied from 12 to 24 months.

The evaluation used the following data sources:

• Volumetric traffic surveys, a twelve-round traffic dataset including vehicle counts and speeds.
• The Settlement Information Survey, a panel survey of 960 communities conducted in three rounds: a baseline in 2007, a second round in 2010, and a third round in August 2012.
• The Integrated Household Survey, a dataset of household level outcomes collected between the first quarter of 2003 and the fourth quarter of 2011, for a total of 36 rounds.

2020-002-2391