Measuring Results of a Rural Road Rehabilitation Project in Armenia

In Context

The MCC compact with Armenia was a five-year investment (2006- 2011) of originally $236 million in two projects: (i) the Rural Road Rehabilitation Project (RRRP) ($67 million) and (ii) the Irrigated Agriculture Project ($169 million). As a result of post-election events in March 2008 that were inconsistent with MCC’s principle of promoting democratic governance and related declines in Armenia’s performance on MCC’s Ruling Justly indicator, MCC placed a hold on funding for the compact’s RRRP. Approximately 24 kilometers of pilot roads were improved under the MCC project using $8.4 million before the hold was placed by MCC; however no additional road construction was done under the MCC program. In 2009, the Armenian government began accessing loans from the World Bank to rehabilitate many road sections that were included originally in the RRRP plans. Between 2009 and 2013, the World Bank financed $100 million of road rehabilitation for 430 kilometers under the Lifeline Road Improvement Project primarily based on the MCC road designs.

As part of the compact Monitoring & Evaluation (M&E) Plan, MCC funded an increase in the sample size of the Armenia national household survey (the Integrated Survey of Living Standards or ISLS) conducted by the National Statistical Service of Armenia on an annual basis. MCC’s independent evaluators planned on using the ISLS to evaluate the impact of RRRP before the project was placed on indefinite hold. Since the increase in sample of the ISLS was not tied to the RRRP, the data collection continued until the end of the compact and was provided to MCC. Both the Government of Armenia and the World Bank expressed interest in having MCC’s independent evaluator analyze the ISLS data as previously planned to provide insights into the impacts of rural road rehabilitation. In addition, MCC saw value in completing the evaluation because of the large percentage of MCC’s portfolio invested in roads.

Program Logic

The original RRRP was designed to improve the quality of the lifeline road network in Armenia in order to enhance the economic performance of the agricultural sector. It was expected that improved road quality would reduce transportation costs and increase vehicular activity, which would increase access to markets and social infrastructure. As a result, farmers would be able to access agricultural inputs at cheaper prices and an increased number of retailers and buyers of agricultural products could access the communities, thus creating conditions for farmers to sell their agricultural production at a higher price. These changes would incentivize farmers to invest more, thereby increasing employment and production, leading to improved performance of the agricultural sector and poverty reduction.

The World Bank’s program logic was somewhat different from the RRRP’s. Since the World Bank project was approved during the 2009 financial crisis, there was an emphasis on creating short-term employment through road construction. In addition, the World Bank only explicitly targeted the immediate and short-term outcomes related to transportation. Though the medium- and long-term outcomes would hopefully follow from the immediate and short-term outcomes, the World Bank did not state these as goals for the road projects that they financed.

 

There were several key assumptions underlying the program logic during the design of the investment:

  • Farmers and agricultural traders would change behavior as a result of improved road quality and lower transport costs
  • Agricultural input prices and crop sales prices would be affected by lower transport costs

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

The following table summarizes performance on output and outcome indicators specific to the evaluated programs:

Monitoring Indicators Tracked During Implementation of the Rural Road Rehabilitation Project

Indicators Level Baseline Actual Achieved Target Percent Complete
Average daily traffic on pilot roads Outcome 637 735 706 142%
International roughness index for pilot roads Outcome 14.16 3.47 4 105%
Government budgetary allocations for rehabilitation of road sections in the road lifeline network 1 Outcome N/A 7,227,000 3,310,000 218%
Government budgetary allocations for routine maintenance of the entire road network Outcome N/A 6,656,000 6,290,000 106%
Pilot roads rehabilitated Output 0 24.4 24.4 100%
 

(Source: Final Indicator Tracking Table, December 2014.)

Monitoring Indicators Tracked During Implementation of the Rural Road Rehabilitation Project (World Bank Lifeline Road Improvement Project)

Indicators Level Baseline Actual Achieved Target Percent Complete
Percent reduction in travel time Outcome 0 58.5 20 293%
Transport cost reductions on project roads Outcome 0 25.8 20 109%
Number of person- days/month of jobs created Output 0 39,855 36,6506 109%
Roads rehabilitated Output 0 446 430 104%
 

(Source: The World Bank’s Implementation Completion and Results Report. http://www- wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2014/ 06/19/000333037_20140619121331/Rendered/PDF/ICR31450P115480C0 disclosed060170140.pdf)

The average completion rate of output and outcome targets for the MCC Armenia road project is 134% percent; and for 5 of the 5 indicators, targets were met or exceeded. However, these calculations are based on revised targets after the project was placed on hold indefinitely. The original target was 943 kilometers to be rehabilitated, so 2.6% of the original target was met before the project was placed on hold. Based on the subset of indicators presented here, the average completion rate of output and outcome targets for the World Bank Armenia road project is 159%. The World Bank also calculated an ex- post ERR of 18.1%, only slightly lower than the ex-ante ERR of 18.4%.

Evaluation Questions

The evaluation of the RRRP aimed to answer the following evaluation questions:

  • Did rehabilitating roads affect the quality of roads?
  • Did rehabilitating roads improve access to markets and social services?
  • Did rehabilitating roads improve income from employment?
  • Did rehabilitating roads affect agricultural productivity and profits, and if so, by how much?
  • Did rehabilitating roads improve household well-being for communities served by these roads, especially income and poverty?

 

Evaluation Results

The evaluation compares 27 road sections that were originally designed by the MCC Armenia program and then financed by the World Bank with 28 road sections that were originally included in the MCC Armenia program, but never rehabilitated. The findings imply that road rehabilitation efforts can improve road quality and increase the use of roads in the short term, but that they may not be sufficient to stimulate increases in agricultural production and sales in a time frame of one to two years.

Rural Road Rehabilitation Project – Pilot Roads
Evaluator Mathematica Policy Research
Evaluation Type Impact
Methodology Difference-in- differences
Exposure Period 2007 and 2008 (Baseline); 2011 (Endline). Most roads were completed in 2009-2010 for an exposure period of 1 to 2 years after rehabilitation.
Immediate Outcomes
  • Improved road quality – 39 percentage point increase in favorability rating of regional roads
  • Employment linked to construction – inconclusive 2
  • Reduced travel time – strong indirect evidence of large impacts
Short term Outcomes
  • Improved access to markets – 20 percentage point decrease in market access difficulties
  • Improved access to social services – no evidence of impacts
  • Increased vehicular activity – 17 percentage point increase in use of roads for noncommercial purposes
Medium term Outcomes
  • Investment – limited evidence of small impacts
  • Employment – no evidence of impacts
  • Production – limited evidence that may be anomalous
Long term Outcomes
  • Household income – no evidence of impacts
  • Household consumption – no evidence of impacts
  • Rural poverty – some evidence of increase in rural poverty; likely an anomaly due to sample composition

Lessons Learned

Several key lessons learned from this evaluation contribute to a broad set of lessons derived from other roads evaluations and the findings of the Transport Sector Practice Group’s internal reviews. MCC and partner countries should consider these lessons when designing and implementing roads projects and evaluations.

Improving evaluation methodology:

  • Base evaluation decisions on a clear program logic. The Armenia roads rehabilitation was designed to improve the economic performance of the agricultural sector; however, the initial program logic was fairly vague about the pathways through which this would occur. In addition, there was limited evidence available on which to base the original program logic. This made the evaluation design more challenging as the specific outcomes to be assessed were defined in general terms, and there were no clear theories about the expected timing of changes in those outcomes. The decision of when and what data should be collected should be driven by a clear program logic that underlies the investment decision.
  • Set realistic time horizons and keep data collection plans flexible. Often there are delays in large infrastructure projects. The data collection plan for the Armenia roads project was agreed upon early in the compact with the National Statistical Service of the Republic of Armenia. Data collection proceeded as planned, but due to unforeseen events in Armenia, there were only 1 to 2 years after road rehabilitation before final data collection took place. This limits the learning from the evaluation as it is not clear if the lack of impact on medium and long term outcomes is due to the limited exposure period or if the road rehabilitation really had a negligible impact on household well-being. From the beginning, implementers and evaluators should build into the evaluation design actions for mitigating risk to the evaluation associated with delays in implementation.
  • Ensure sufficient statistical power. The evaluation included a small number of road sections – 27 in the treatment group and 29 in the comparison group. Since there is intra-cluster correlation amongst households located around any one road section, the statistical power of the evaluation was limited. As a result, it is challenging to interpret the results for the medium and long term outcomes. Was there really very little impact or were the impacts just not as large as the minimum detectable effects, which for many medium and long term variables were quite high. The weak statistical power was due to two main factors: (1) a small number of road sections actually being rehabilitated and (2) design of the evaluation around an existing data source which limited the number of road sections covered and the sample size. In the future, before investing significant resources into an evaluation, MCC should ensure that there will be sufficient statistical power for measuring realistic changes in key outcomes. To achieve this goal, it may be productive to for an evaluation to cover similar programs in multiple countries rather than for each evaluation to only examine one country at a time.  Many of these lessons are similar to those that MCC has learned from previous evaluations. As a result, MCC has already adjusted its evaluation practices to include a formal review process for evaluations, an evaluation risk assessment, and use of standardized evaluation templates. The new process also requires substantive review and clearance of key evaluation documents by sector specialists in order to incorporate feedback on the technical and factual accuracy of evaluation plans.
  • A multifaceted development approach requires proactive and visionary management. The Irrigated Agriculture Project suffered from poor integration of project activities and targeting of beneficiaries. Project activities were broken into several different contracts, which increased the challenge of coordination among contractors’ timelines and activities. Mid-course corrections such as improvements in coordination among contractors, implementation strategy and staffing changes reduced the risks inherent in the piecemeal implementation approach, which improved implementation performance. Nonetheless, sequencing challenges compromised the original program logic.
  • The evaluation questions are based on the program logic and must be designed carefully from the beginning to understand the scope and limitations of the evaluation. Given that the WTM Activity was not designed and implemented as a package of coordinated interventions for a targeted group of beneficiaries, MCC could not design an evaluation of the overall WTM Activity. The project design, implementation and the corresponding independent evaluations have limited MCC’s ability to report on the overall impact of the WTM Activity. In the future, MCC should work with all stakeholders to understand the program logic, how the program will be implemented and clarify what the evaluation will be able to answer and not answer from the beginning.

 

Improving roads project selection and design:

  • The roads project selection process should include an upfront national or area-wide road network analysis based on selected criteria such as traffic volume, IRI and other parameters, in order to prioritize potential road investments that are proven to be economically viable.
  • It is important to consider alternative interventions that may prove to be relatively more cost effective and economically viable than simply paving a road.
  • It is critical to comprehensively address policy and institutional constraints in road maintenance as well as seek assurances from the partner countries that the necessary mechanisms to ensure sustainability of their existing roadway network are in place prior to MCC committing to a capital-intensive road investment project.
  • MCC recognizes the need to better understand actual road maintenance practices and their effects on the long-term costs and benefits of roads. Accordingly, MCC is planning a series of country- specific road maintenance studies, which will be used to improve both the economic assessment of road investments and, where feasible, influence actual road maintenance planning and execution in partner countries.
  • Project teams must ensure complete and high quality data is collected both for the Highway Development and Management (HDM-4)/Roads Economic Decision (RED) ERR modeling purposes that feed into project selection and design, and for M&E purposes during and after implementation. The HDM-4/RED models should be based on fully developed feasibility studies that provide accurate cost and time estimates and other reliable technical inputs. They must also be well developed and calibrated at the feasibility study stage and continue to be updated as costs and other design parameters change throughout the construction stages and post-project completion.
  • The value of roads investments can be optimized by enforcing standards for design review by technical experts and quality assurance and control requirements. Roads teams should also consider alternative forms of engineering contracts and project delivery systems that may improve the quality of contractor feasibility, design and supervision.

 

Next Steps

The results of this evaluation will be presented to stakeholders in Armenia in 2015 along with other evaluations from the MCC Armenia Compact. The evaluation and associated supporting technical documentation, will be available on the MCC external web site for public access thereafter; refer to the MCC Evaluation Catalog available at: http://data.mcc.gov/evaluations/index.php/catalog.

Footnotes
  • 1. The two budgetary indicators were only reported to MCC until 2009 (covering 2008 data) because then the project was placed on operational hold.
  • 2. Note that according to the World Bank, the project generated 39,855 person-days/month of temporary construction jobs compared to a target value of 36,650 person-days/month, of whom 60- 70 percent were hired from local villages and 70 percent unskilled. Such increase corresponds to direct income transfer to workers of almost US$5.0 million.