Measuring Results of the Georgia Samtskhe-Javakheti Road Rehabilitation Activity

The Samtskhe-Javakheti Road Rehabilitation (SJRR) Activity was part of the Regional Infrastructure Rehabilitation Project and represented 51 percent of the total compact. The 220 kilometer Samthskhe-Javakheti (SJ) road aimed at addressing the economic isolation of the Samtskhe-Javakehti region from the capital city of Tbilisi and the rest of the country. The SJRR impact evaluation compares segments of the Samtskhe-Javakheti road which were rehabilitated by MCC funds, with road segments which were not rehabilitated to estimate the impacts of the road rehabilitation on household- and village-level outcomes.  There is strong evidence that the road rehabilitation increased the number of vehicles using the roads and the speed of those vehicles.  Intermediate outcomes suggest some improvements in the economic conditions in the areas near the road, such as increased industrial activity and employment, yet direct changes in household-level welfare were not detected, likely due to the short time frame of post-completion data collection and weaknesses in the household data collection. 

In Context

Pie Chart: Distribution of Funding in Georgia I Compact

 

The Government of Georgia (GoG) and Millennium Challenge Corporation (MCC) signed a $295.3 million Compact in 2005 that aimed to reduce poverty through economic growth by focusing on key constraints to development through rehabilitation of dilapidated infrastructure, improvements to roads and energy infrastructure, and investment in SMEs and agribusinesses. An amendment to the Compact was signed on November 20, 2008 on provision of additional $100 million assistance.  The compact was composed of two projects:  the Regional Infrastructure Rehabilitation Project and the Enterprise Development Project. The Regional Infrastructure Rehabilitation Project included three major activities, Samtskhe-Javakheti Road Rehabilitation (SJRR) Activity, Regional Infrastructure Development Activity, and the Main Gas Pipeline Rehabilitation Activity. The Samtskhe-Javakheti Road Rehabilitation Activity represents 51 percent of the total compact costs, at $203.5 million.

Program Logic

The Regional Infrastructure Rehabilitation Project was designed to rehabilitate key portions of dilapidated infrastructure, with the rehabilitation of the Samthskhe-Javakheti (SJ) road aimed at addressing the economic isolation of the Samtskhe-Javakehti region from the capital city of Tbilisi and the rest of the country.

Visualization of Program Logic

There were several key assumptions underlying the SJRR program logic during the design of the investment:

  • The S-J road rehabilitation would complement other road development projects
  • Agricultural output across all sectors would increase 10 percent in 2010 and, through 2020, rise to the levels experienced during the Soviet era

 

Measuring Results

MCC uses multiple sources to measure results. Monitoring data is used during compact implementation.  Independent evaluations are generally completed post-compact. Monitoring data is typically generated by the program implementers, and specifically covers the program participants who received treatment through the compact. However, monitoring data is limited in that it cannot tell us what these program participants would have done in the absence of the MCC-funded training. This is a key motivation for why MCC invests in independent impact evaluations, which estimate a counterfactual to assess what would have happened in the absence of the investment.

 

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program, of which 4 of 5 key indicators were fully met or achieved during the life of the compact.

 

Indicators

Level

Baseline

Achieved

(Sept 2011)

Target

Percent Complete

Savings in vehicle operating costs (VOC)

Outcome

0

13,770,000

13,177,000

104.5%

International Roughness Index (lower number = smoother road)

Outcome

16.6

1.5

2.5

107%

Annual Average Daily Traffic (Vehicles)

Outcome

612

1,092

1,183

84%

Travel Time (Hour and minute)

Outcome

8:13

2:42

2:45

102%

Kilometers of road upgrades completed (7 contracts)

Output

0

220

220

100%

 

 

Evaluation Questions

The evaluation was designed to answer the following top-level question about the activity: “How does the road rehabilitation effect/cause economic development, new businesses, and economic and social integration in the region?”

 

In addition, the following categories of outcomes were explored by the evaluation:

  • Transportation related outcomes: traffic counts, vehicle speeds, travel times, and availability of public transport
  • Investment, land use, and employment: industrial investment, land uses, cropping patterns, employment
  • Market prices: the prices of basic commodities on the local market
  • Household welfare: income, consumption, asset ownership
  • Access to health and education: utilization of health care and education services

 

Evaluation Results

The SJRR impact evaluation compares segments of the Samtskhe-Javakheti road which were rehabilitated by MCC funds, with road segments which were not rehabilitated to estimate the impacts of the road rehabilitation on the abovementioned household- and village-level outcomes.  There is strong evidence that the road rehabilitation increased the number of vehicles using the roads and the speed of those vehicles.  Intermediate outcomes suggest some improvements in the economic conditions in the areas near the road, such as increased industrial activity and employment, yet direct changes in household-level welfare were not detected, likely due to the short time frame of post-completion data collection and weaknesses in the household data collection.

 

 

Evaluator            

NORC at the University of Chicago

Methodology   

-Difference in Difference

-Continuous Treatment

-Matched Difference in Difference (matching counterfactual comparison road section using propensity score matching)

Evaluation Period

Spring 2008 – December 2010

Immediate Outcome

  • The volume of vehicles on project roads increased by an average of 44.2 vehicles per day (4.2%) with respect to comparison roads
  • The average speed along the roads increased by 13.6 km/h (24.4%) with respect to comparison roads.

Intermediate Outcome

  • Strong evidence that S-J road improvement led to a 13.4% increase in the number of industrial facilities (i.e. canneries, factories, agricultural processing facilities, and similar enterprises) in settlements near the project roads
  • There is some evidence of an increase in the household employment rate in project areas by an estimated 3.75%
  • Local prices were affected in complex ways (some increased, some decreased), whose interpretation is unclear

Ultimate Impact

  • No impacts were detectable on income, consumption, asset ownership, or utilization of health and education services at the household level
 

 

As with any quasi-experimental evaluation of a roads investment, it is important to bear in mind that when comparing the S-J road rehabilitation with comparison roads in other parts of the country, the results could be driven by differences between the treatment and comparison groups. Additionally, the timeframe over which outcomes were measured was less than two years after the conclusion of the project, thus it is possible that these results are unable to capture the full impact of the project.

 

Lessons Learned

Several key lessons learned from this evaluation contribute to a broad set of lessons derived from other roads evaluations and the findings of the Transport Sector Practice Group’s internal reviews.  MCC and partner countries should consider these lessons when designing and implementing roads projects and evaluations:

 

Improving evaluation methodology in roads projects:

 

Base evaluation decisions on a clear program logic. The SJ Roads Rehabilitation was designed to improve the economic performance of the agricultural sector in the SJ region; however, the initial program logic was fairly vague about the pathways through which this would occur. This made the evaluation design more challenging as the specific outcomes to be assessed, populations expected to benefit, and expected timelines of benefits were all defined in general terms. The decision of when and what data should be collected should be driven by a clear program logic that underlies the investment decision.

 

Set realistic time horizons and keep data collection plans flexible. Inevitably there is major potential for delays in large infrastructure projects. The timing of data collection efforts should match the actual implementation of the project, as collecting endline data too soon after project completion limits the learning from the evaluation.  For example, on a number of dimensions in the SJ road rehabilitation evaluation, it is not clear if the lack of impact on medium and long term outcomes is due to the limited exposure period or if the road rehabilitation really had a negligible impact on household well-being (e.g. income, consumption, asset ownership, and utilization of health and education services). From the beginning, implementers and evaluators should build into the evaluation design actions for mitigating risk to the evaluation associated with delays in implementation.

 

Ensure sufficient statistical power.  In the future, before investing significant resources into an evaluation, MCC should ensure that there will be sufficient statistical power for measuring realistic changes in key outcomes.  Measuring many of the Samtskhe-Javakheti Road Rehabilitation’s intermediate outcomes and ultimate impacts proved difficult, given that impacts are generally disperse (i.e. covering large populations).  With a better understanding of the expected impacts and a clearer identification of beneficiaries, a more precise alignment between an intervention’s program logic and its evaluation design can be achieved. In addition, the challenges with using the household data which was collected in measuring impacts suggest that data collection strategies can also be improved and tailored to the given evaluation methodology. 

 

Many of these lessons are similar to those that MCC has learned from previous evaluations. As a result, MCC has already adjusted its evaluation practices to include a formal review process for evaluations, an evaluation risk assessment, and use of standardized evaluation templates.  The new process also requires substantive review and clearance of key evaluation documents by sector specialists in order to incorporate feedback on the technical and factual accuracy of evaluation plans.

 

Improving roads project selection and design:

 

Ensure upfront network analysis: The roads project selection process should include an upfront national or area-wide road network analysis based on selected criteria such as traffic volume, IRI and other parameters, in order to prioritize potential road investments that are proven to be economically viable. 

 

Ensure institutional and policy reforms are implemented: As Georgia is one of MCC’s earlier compacts, the policy and institutional reform aspects of the road project may not have been as strong as the later compacts, where these issues were given more serious consideration for investment decisions. It is critical to comprehensively address policy and institutional constraints in road maintenance as well as seek assurances from the partner countries that the necessary mechanisms to ensure sustainability of their existing roadway network are in place prior to MCC committing to a capital-intensive road investment project.  MCC recognizes the need to better understand actual road maintenance practices and their effects on the long-term costs and benefits of roads.  Accordingly, MCC is planning a series of country-specific road maintenance studies, which will be used to improve both the economic assessment of road investments and, where feasible, influence actual road maintenance planning and execution in partner countries. 

 

Consider alternative intervention investments:  The justification to perform some type of rehabilitation on the SJ road was strong given the very poor condition of the road (i.e. IRI of 16.6), which resulted in high vehicle operating costs and long travel times.  When making future roads investments, MCC is taking a closer look at alternative interventions for road investments that may prove to be relatively more cost effective and economically viable than simply fully re-paving a road.  These alternatives range from targeted spot improvements to consideration of periodic maintenance projects much more in line with the projected traffic levels and levels of service of the road.

 

Optimize value of road investment through better design and quality control during construction:  Limited sections of the SJ road have experienced premature pavement failure after rehabilitation during the Defects Notification Period.  The potential causes of such failure were traced to design issues and as well as to lapses in quality control during construction.  Establishing and enforcing an MCC standard for design review, which incorporates a greater level of technical expertise, could help to identify possible design deficiencies early on in the project development phase.  In addition to more rigorous oversight processes, it is important to strongly enforce quality assurance and control requirements, as well as consider alternative forms of engineering contracts and project delivery systems to improve the quality of contractor feasibility, design and supervision.

 

Ensure investment decisions are made based on reliable technical and cost data:  Significant cost increases were experienced on a per kilometer basis in the SJ Road Rehabilitation project, notably from the periods between funding authorization and the final engineer’s estimate, and  contract award and final cost after completion; this had a negative effect on the overall return of the investment.  Such increases can be attributed to a number of factors, but are mainly due to timeline pressure that resulted in investment decisions being made without the benefit of fully developed feasibility studies and final design cost data.  The need to establish and enforce an MCC standard for complete feasibility study content, quality and timing prior to project funding authorization is one of the critical lessons learned from the early road projects at MCC, including the SJ Road Rehabilitation project.

 

Continue to refine the use of investment decision models:  The cost benefit analysis that underlies MCC’s investment decisions on roads projects relies on the quality and completeness of road economic models, such as the Highway Development and Management (HDM-4) Model or the Roads Economic Decision (RED) Model.  It is important to ensure that complete and high quality data is collected both for HDM-4/RED ERR modeling purposes that feed into project selection and design and for M&E purposes during and after implementation.  The HDM-4/RED models must be well developed and calibrated at the feasibility study stage and continue to be updated as costs and other design parameters change throughout the construction stages and post-project completion.