Measuring Results of the Nicaragua Transportation Project

In Context

The MCC compact with Nicaragua was a five-year investment (2006 – 2011) of $112.7 million in three projects: (i) Transportation Project, (ii) Property Regularization Project and (iii) Rural Business Development Project 1. The Transportation Project consisted of the upgrading of one secondary trunk road and two secondary roads totaling 68 kilometers in length at a cost of $57.9 million, representing 51 percent of the total compact. The $57.9 million allocated to the Transportation Project is the subject of both the results described here and an independent impact evaluation conducted by Dr. Jonathan E. Alevy and released by MCC in early 2015.

Nicaragua was one of the first countries to be selected for a compact with MCC in 2004. The compact was signed in 2005 and entered into force in 2006. The Government of Nicaragua (GON) identified a regional development strategy to address with MCC support the challenges of insecure property rights, underdeveloped infrastructure and low-value agriculture production. The compact focused on creating an engine for economic growth in the northwestern part of the country, a region with significant growth potential due to its fertile land and connection to markets in Honduras and El Salvador.

Program Logic

Under the Transportation Project, MCA-Nicaragua substantially upgraded and rehabilitated 68 km of roads: two secondary roads – Somotillo-Cinco Pinos (S1) in the north and León-Poneloya-Las Peñitas (S9) linking the urban center of León to oceanfront communities, and a secondary trunk road – Villanueva-Guasaule (V-G) connecting the northern city of Villanueva to the Honduran border at El Guasaule. The rehabilitation included resurfacing and extensive drainage and grading improvements. Benefits of the road upgrades were expected to accrue to users of the road in the form of decreased vehicle operating costs and decreased travel time; benefits were expected to accrue to communities living within the zone of influence of the road upgrades in the form of lower prices and increased availability of consumer goods.

 

There were several key assumptions underlying the program logic during the design of the investment:

  • Traffic increases at a rate of about 6 percent per year
  • Return on the secondary roads will be at least 8 percent
  • Economic benefit is derived from reduced vehicle operating costs and travel time
  • Investment, maintenance and transport cost savings capture all the major costs and benefits of a project
  • Prices of consumer goods respond to transportation costs
  • The poor will benefit from reduced travel time on their daily commute, which will allow them to use the time savings for other productive activities
  • Benefits would begin to accrue shortly after the completion of the road rehabilitation

 

Measuring Results

MCC uses multiple sources to measure results. Monitoring data is used during compact implementation. Independent evaluations are generally completed post-compact. Monitoring data is typically generated by the program implementers within the 5-year time frame of implementation, and specifically covers the program participants who received treatment through the compact. However, monitoring data is limited in that it cannot tell us what these program participants would have done in the absence of the MCC-funded investment. This is a key motivation for why MCC invests in independent impact evaluations, which estimate a counterfactual to assess what would have happened in the absence of the investment.

The following table summarizes performance on output and outcome indicators specific to the evaluated Transportation Project:

Monitoring Indicators Tracked During Implementation of the Transportation Project

Indicators Level Actual Achieved Target Percent Complete
Average Annual Daily Traffic – Villanueva – Guasaule (V-G) Outcome 1,961 1,580 328%
Average Annual Daily Traffic – Somotillo-Cinco Pinos (S1) Outcome 561 278 743%
Average Annual Daily Traffic – León-Poneloya-Las Peñitas (S9) Outcome 1,462 1,276 208%
International Roughness Index – Villanueva – Guasaule (V-G) Outcome 1.76 3.4 119%
International Roughness Index – Somotillo-Cinco Pinos (S1): Outcome 3.38 3 96%
International Roughness Index – León-Poneloya-Las Peñitas (S9) Outcome 1.84 3 113%
Kilometers Upgraded – Villanueva – Guasaule (V-G) Output 18 18 100%
Kilometers Upgraded – Somotillo-Cinco Pinos (S1) Output 29.4 29.4 100%
Kilometers Upgraded – León-Poneloya-Las Peñitas (S9) Output 19.6 19.6 100%
 

 

Note that Average Annual Daily Traffic ‘actuals achieved’ presented in this table differ from those found in the evaluation, the figures presented in this table are sourced from a local data collection firm whereas the evaluator used figures from the Ministry of Transportation and Infrastructure.

“Percent Complete” is calculated using the formula (Actual Achieved – Baseline) / (Target – Baseline); this captures the movement of the Actual towards the Target with respect to the Baseline.

Evaluation Questions

The evaluation of the Transportation Project aimed to answer the following questions:

  • Did the rehabilitation of roads affect the quality of roads?
  • Did the rehabilitation of roads reduce vehicle operating costs?
  • Did the rehabilitation of roads affect the price of goods?
  • Did the rehabilitation of roads affect the availability of goods?

 

Evaluation Results

To evaluate the impact of the MCA Nicaragua Transportation Project, the Independent Evaluator used two methods: (i) the calculation of ex-post economic rates of return (ERRs) for the individual roads and the Transportation Project as a whole and (ii) difference-in-difference estimates of the availability and price of goods at retail establishments both within and outside of the zone of influence.

The calculation of the ex-post ERR utilized the Roads Economic Decision model (RED) developed by the World Bank. This approach captures the project’s flow of net benefits to road users defined as benefits minus costs which are discounted across time to yield a single value, using pre- and post-construction traffic counts and estimates of future traffic counts. The ex-post ERR estimated by the independent evaluator showed that the roads failed to meet the 10 percent hurdle rate. The average for the project, as a whole, was estimated to be 2.1 percent. Actual capital costs were, on average, 2.2 times greater than those estimated in the feasibility studies.

The Independent Evaluator sought to determine how local economies respond to lower transportation costs through a survey of retail establishments using a double difference comparison. Data was collected from establishments within the zone of influence of the improved roads (treatment) and in a group of comparable establishments outside the zone of influence of the improved roads (comparison) before and after the road upgrades. The sample frame was selected by identifying similar communities in and out of the project area using propensity score matching (PSM) techniques and then randomly selecting establishments within each community. The survey collected information from retail establishments regarding prices and availability of goods included in the Canasta Basica, or basic basket of goods, that is used in Nicaragua to track consumer prices. The impact of the roads on the value of the basic basket of goods is close to zero in both urban and rural areas. The availability of goods increased in both project and non-project communities, and although there is a slightly larger increase in project communities it is not a statistically or economically significant effect.

 
Evaluator Dr. Jonathan E. Alevy
Evaluation Type Performance
Methodology Independent Ex-Post ERR using modified RED model
Exposure Period All road upgrades were completed in early 2010 with traffic counts taking place in 2012, for an exposure period of less than two years.
Ex-post ERR The ERR calculations reveal that the roads fail to meet the 10% hurdle rate for ERR with mean estimates as follows:

  • Villanueva – Guasaule (V-G): 3.8%
  • Somotillo-Cinco Pinos (S-1): -3.9%
  • León-Poneloya-Las Peñitas: 4.5%

The overall project average ERR is 2.1%.

 
Evaluator Dr. Jonathan E. Alevy
Evaluation Type Impact
Methodology Difference-in-Difference with Matching. The evaluation approach is a double difference comparison. The double difference is comprised of:

  1. First difference: comparing prices and availability of goods in communities affected before and after the upgrading (treatment).
  2. Second difference: comparing prices and availability of goods in communities NOT affected by the upgrading (comparison) before and after the upgrading to those communities in point 1 above (treatment).

Matching – The sample for the establishment survey was created using propensity score matching methods (PSM). The comparison group was chosen from communities on roads considered for upgrading and in which project feasibility studies were conducted. The matching methods identified the communities most similar to those in the treatment group based on characteristics that included population, initial road quality, and share of population with electric lighting.

Exposure Period August 2008 (Baseline); October 2010 (Endline). All road upgrades were completed in early 2010, for an exposure period of less than one year.
Impact The impact on the value of the basic basket of goods is close to zero in both urban and rural areas. In rural areas there is a small decline (0.97 percent) and in urban areas there is a small increase (0.91 percent) in the value of the basic basket of goods, although it is noted that this price increase is being driven by the relatively large price increase of one good (detergent). The availability of goods increased in both project and non-project communities, and although there is a slightly larger increase in project communities (2.55 additional items available in project communities) it is not a statistically or economically significant effect.

Lessons Learned

Several key lessons learned from this evaluation contribute to a broad set of lessons derived from other roads evaluations and the findings of the Infrastructure Practice Group’s internal reviews. MCC and partner countries should consider these lessons when designing and implementing roads projects and evaluations.

Improving evaluation methodology:

  • Investment decisions, and the ERRs informing them, should be based on final design cost estimates. Prior cost estimates and designs based on pre-feasibility studies often led to overly risky investment decisions.
  • Safety benefits were not taken into consideration in the calculation of the post-compact ERRs. In the future, and to the extent that safety-related information is available, MCC plans to incorporate the economic benefits of road safety into ERR calculations when the needed information is available.
  • The decision of when and what data should be collected should be driven by the logic of the investment and a good critical sense of how to gather information cost-effectively. Some markets may respond gradually to improved road conditions, therefore, using traffic counts collected less than two years after road completion may not allow sufficient time to detect important effects of road improvements. As MCC designs transportation projects and their evaluations, outcome measures should be carefully chosen to cost-effectively and credibly assess the stated economic justification of road investments.
  • Plans for post-investment road ERR evaluations should be subjected to a formal review process that requires (among others) substantive review and clearance by sector specialists of key evaluation documents to provide feedback on the technical and factual accuracy of evaluation plans. This approach should ensure that the selection of the Highway Development and Management (HDM-4) vs. RED models to estimate the long-term costs and benefits of the road investments is appropriate to the characteristics of the road to be evaluated.
  • Many of these lessons are similar to those that MCC has learned from previous evaluations. As a result, MCC has already adjusted its evaluation practices to include a formal review process for evaluations, an evaluation risk assessment, and use of standardized evaluation templates. The new process also requires substantive review and clearance of key evaluation documents by sector specialists in order to incorporate feedback on the technical and factual accuracy of evaluation plans.

 

Improving roads project selection and design:

  • The roads project selection process should include an upfront national or area-wide road network analysis based on selected criteria such as traffic volume, IRI and other parameters, in order to prioritize potential road investments that are proven to be economically viable.
  • It is important to consider alternative interventions that may prove to be relatively more cost effective and economically viable than simply paving a road.
  • It is critical to comprehensively address policy and institutional constraints in road maintenance as well as seek assurances from the partner countries that the necessary mechanisms to ensure sustainability of their existing roadway network are in place prior to MCC committing to a capital-intensive road investment project.
  • MCC recognizes the need to better understand actual road maintenance practices and their effects on the long-term costs and benefits of roads. Accordingly, MCC’s Infrastructure Practice Group is planning a series of country-specific road maintenance studies, which will be used to improve both the economic assessment of road investments and, where feasible, influence actual road maintenance planning and execution in partner countries.
  • Project teams must ensure complete and high quality data is collected both for HDM-4/RED ERR modeling purposes that feed into project selection and design, and for M&E purposes during and after implementation. The HDM-4/RED models should be based on fully developed feasibility studies that provide accurate cost and time estimates and other reliable technical inputs. They must also be well developed and calibrated at the feasibility study stage and continue to be updated as costs and other design parameters change throughout the construction stages and post-project completion.
  • The value of roads investments can be optimized by enforcing standards for design review by technical experts and quality assurance and control requirements. Roads teams should also consider alternative forms of engineering contracts and project delivery systems that may improve the quality of contractor feasibility, design and supervision.

 

Next Steps

The evaluation Final Report in conjunction with anonymized data sets and associated supporting technical documentation, will be available on the MCC external web site for public access and use in early 2015; refer to the MCC Evaluation Catalog available at: http://data.mcc.gov/evaluations/index.php/catalog.

Footnotes
  • 1. Conditions leading up to, during and following municipal elections of November 2008 were inconsistent with MCC’s eligibility criteria. In June 2009, MCC’s Board terminated a portion of the compact, reducing compact funding from $175 million to $113.5 million. Funding was terminated for all activities in the Property Regularization Project and for activities in the Transportation Project which were not already under contract; the Rural Business Development Project was not affected. Under the Transportation Project the planned rehabilitation of the 58-km stretch of road between Izapa and Nejapa on the outskirts of Managua was suspended (N-1 Corridor).