Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

  • Final Evaluation Brief
  • January 2013

Rehabilitating the Samtskhe-Javakheti Road in Georgia

Travel conditions improved, but household impacts were not observed

View as PDF

Program Overview

MCC’s $395.3 million Georgia Compact (2006-2011) funded the $203.5 million Samtskhe-Javakheti (SJ) Road Rehabilitation Activity to address the SJ region’s economic isolation from the capital city of Tbilisi and the rest of the country. The activity aimed to reduce the time and costs associated with transportation by rehabilitating and constructing 220 kilometers of road. The improved road was expected to increase traffic volume, reduce travel times, and lower prices of consumer goods, ultimately leading to increased household consumption and higher levels of business investment.

Evaluator Description

MCC commissioned NORC at the University of Chicago to conduct an independent final impact evaluation of the Samtskhe-Javakheti Road Rehabilitation Activity. Full report results and learning:

Key Findings

Travel and Transport

  • Rehabilitating the Samtskhe-Javakheti (SJ) road successfully improved travel conditions. Traffic volume and average speed increased, and travel times to both the capital city of Tbilisi and local markets decreased.

Industrial Investment, Land Use, and Employment

  • The improved roads significantly increased industrial investment in nearby communities.
  • There were no observable changes in cropping patterns or agriculture land use at the community level.

Market Prices

  • Local prices of commodities were affected by the improved roads, but in ways that were difficult to interpret in relation to the program’s objectives.

Household Welfare

  • The evaluation did not find that the improved roads impacted income, consumption, asset ownership, employment, or utilization of health and education services at the household level.
  • However, the timing of the evaluation, which was less than two years after the project concluded, limited its ability to assess impacts on household welfare.

Evaluation Questions

This final impact evaluation was designed to answer the following top-line question:

  1. 1
    How does the road rehabilitation affect/cause economic development, new businesses, and economic and social integration in the region?
  2. In addition, the following categories of outcomes were explored by the evaluation:

    • Transportation related outcomes: traffic counts, vehicle speeds, travel times, and availability of public transport
    • Investment, land use, and employment: industrial investment, land uses, cropping patterns, and employment
    • Market prices: the prices of basic commodities on the local market
    • Household welfare: income, consumption, and asset ownership
    • Access to health and education: utilization of health care and education services.

A poor quality segment of the Samtskhe-Javakheti Road before rehabilitation.

Detailed Findings

Reader’s note: In interpreting these findings, it is important to bear in mind some limitations of the analysis. First, while the methodology corrects for selection bias to the greatest extent possible, the results could be driven by differences between the treatment and comparison groups. Additionally, the timeframe over which outcomes were measured was less than two years after the conclusion of the project. Thus, it is possible the results were unable to capture the full impact of the project on outcomes.

Travel and Transport

The estimated International Road Roughness Index for project road segments decreased from 16.6 to 1.5 after treatment.

After construction was completed, the volume of traffic on project roads increased by an average of 44.2 vehicles per day (4.2 percent) relative to comparison roads, while the average speed along the roads increased by 13.6 km/h (24.4 percent). Additionally, there were large and significantly reduced self-reported travel times both to Tbilisi and to local markets where farmers sell their products. The difference-in-difference estimate indicated a reduced driving time to Tbilisi of 39.6 minutes and a 43.6 minute reduced driving time to local markets, while the continuous treatment model finds that a community located 30 minutes from the treatment road would experience a 83.6 minute reduction in travel time to Tbilisi and a 25.9 minute reduction to the local market.

Graphic of travel and transport results after road construction

Industrial Investment, Land Use, and Employment

26.9% increase in the number of industrial facilities in settlements near the project roads.

At the community level, the improved roads in the region led to an estimated 26.9 percent increase in the number of industrial facilities (i.e. canneries, factories, agricultural processing facilities, and similar enterprises) in settlements near the project roads, although some of the impact may be due to selection bias. There were no observable changes in employment outcomes or in agricultural outcomes, such as cropping patterns or land use.

Market Prices

The evaluation found a complex relationship between prices and transport costs. In many cases, statistically significant findings were observed, but with different tendencies for different products, potentially related to depending on whether or not the product in question was produced in the local area. The strongest findings were for honey and beef. The project tended to cause honey prices to increase in areas where honey was produced and decrease prices in more distant markets. The opposite tendency was observed for beef; improving the road tended to lead to a fall in prices where beef was produced and a rise in prices of beef elsewhere.

Household Welfare

There was no evidence of impacts on household-level outcomes in the SJ region, such as income, consumption, or utilization of health and education services. One possible explanation is that the improved SJ roads did not impact these outcomes. However, in light of the significant findings observed at the community level, a more likely explanation is that impacts were not detected due to limitations of the household dataset, which was not ideally structured for the evaluation. Furthermore, insufficient time may have passed between the concluded construction and data collection for these outcomes to have begun to materialize.

MCC Learning

  • Base evaluation decisions on a clear program logic. The decision of when and what data should be collected should be driven by a clear program logic that underlies the investment decision.

  • Set realistic time horizons and keep data collection plans flexible. From the beginning, implementers and evaluators should build actions to mitigate risks associated with project delays into the evaluation design.

  • Ensure sufficient statistical power to measure realistic changes in key outcomes.

  • Include an upfront national or area-wide road network analysis based on selected criteria such as traffic volume, roughness index, and other parameters in the roads project selection process, as noted in the Principles into Practice: Lessons from MCC’s Investment in Roads.

  • Comprehensively address policy and institutional constraints to road maintenance and seek assurances from the partner government that the necessary mechanisms to ensure sustainability are in place prior to MCC’s investment.

Evaluation Methods

An improved segment of the Samtskhe-Javakheti Road.

The evaluation used three methodologies to rigorously evaluate the causal impact of the program on outcomes. The first is a difference-in-difference methodology, whereby the project roads were matched to a set of similar comparison roads where no intervention had taken place. The difference-in-difference analysis compared traffic counts and socioeconomic outcomes for residents of communities located near the project roads and residents of communities located near the comparison roads. Secondly, the evaluation incorporated a continuous treatment approach to model project impact in a dose-response framework, so that communities located near project roads were assumed to experience greater impacts than those more distant. Finally, the evaluation estimated a matched difference-in-difference model, using propensity score matching to improve the comparability between treatment and comparison groups. The exposure period varied from 12 to 24 months.

The evaluation used the following data sources:

  • Volumetric traffic surveys, a twelve-round traffic dataset including vehicle counts and speeds.
  • The Settlement Information Survey, a panel survey of 960 communities conducted in three rounds: a baseline in 2007, a second round in 2010, and a third round in August 2012.
  • The Integrated Household Survey, a dataset of household level outcomes collected between the first quarter of 2003 and the fourth quarter of 2011, for a total of 36 rounds.