The MCC compact with Honduras was a $205 million, five-year investment (2005-2010) in two projects: (i) the Rural Development Project and (ii) Transport Project. The Rural Development Project included four activities: (i) farmer training and development, (ii) farmer access to credit, (iii) farm to market roads, and (iv) agricultural public goods grant facility. The Transport Project comprised three major activities, including the Highway CA-5 construction, Secondary Roads construction and rehabilitation, and a Weight Control System.
The Farm to Market Roads Activity investments are equivalent to 30 percent of the Rural Development Project and the Farm to Market Roads and Transport Project investments are 67 percent of the total compact investment. The $138.1 million allocated to Farm to Market Roads and the Transport Project is the subject of both the results described here and an independent evaluation [LINK] conducted by the NORC Independent Evaluator and released by MCC in May 2014.
In 2005, Honduras became the second country to sign a compact with MCC. Low agricultural productivity and high transportation costs were identified as key constraints to economic growth, limiting Honduras’ ability to take advantage of its strategic location. The compact was designed to help small-scale farmers become small-scale entrepreneurs through training to improve their productivity, access to new markets, and access to credit. It was also expected to reduce transportation costs through improvements in road networks to enhance market access and foster greater market integration.
The Transport Project and Farm to Market Roads Activity were designed to reduce transportation costs between targeted production centers and national, regional and global markets. The initial scope called for rehabilitating two major sections of Highway CA-5, upgrading and paving at least 70 kilometers of secondary roads, and developing a vehicle weight control system. Under the Rural Development Project, MCA-Honduras sought to upgrade and pave 600 kilometers of rural roads (farm-to-market roads). Due to increases in costs and a partial re-scoping of the road rehabilitation component of the project, only 65 kilometers of secondary roads and 495 kilometers of rural roads were ultimately upgraded, and the vehicle weight control system was removed from the investments.
There were several key assumptions underlying the program logic during the design of the investment:
- For expected improvement in outcomes related to improved access to wage employment, health, and education services, the main assumption is that there would be access to transport vehicles
MCC uses multiple sources to measure results. Monitoring data is used during compact implementation. Independent evaluations are generally completed post-compact. Monitoring data is typically generated by the program implementers, and specifically covers the program participants who received treatment through the compact. However, monitoring data is limited in that it cannot tell us what these program participants would have done in the absence of the MCC-funded investment. This is a key motivation for why MCC invests in independent impact evaluations, which estimate a counterfactual to assess what would have happened in the absence of the investment.
The following table summarizes performance on output and outcome indicators specific to the evaluated programs:
|Indicators||Level||Actual Achieved||Target||Percent Complete|
|Highway CA-5 T3 International Roughness Index (IRI)||Outcome||2.2||1.9||84.2|
|Highway CA-5 T4 International Roughness Index (IRI)||Outcome||3.2||1.9||31.6|
|Secondary Roads International Roughness Index (IRI)||Outcome||3.2||2.5||72|
|Highway CA-5 Kilometers of Highway Upgraded||Output||49.5||109||45|
|Secondary Roads Kilometers of Roads Upgraded||Output||65.5||65.5||100|
|Rural Roads Kilometers of Roads Upgraded||Output||495.1||499||99|
The average completion rate of output and outcome targets is 72 percent; and in one of the six indicators, targets were met or exceeded.
As stated in the Final Evaluation Report, the evaluation of the Transport Project and Farm to Market Roads Activity aimed to answer whether or not improved conditions throughout the road network:
- Lowered transport costs and travel time for businesses, including farm households;
- Provided better access to a wider range of job opportunities for individuals (labor market effects);
- Lowered the price of consumables and inputs by increasing competition and reducing barriers to entry posed by poor transport infrastructure; and
- Improved access to health establishments and schools
The overall expected result of these changes was an increase in overall incomes and employment at the household level.
To comprehensively and prospectively evaluate the impact of the MCA Honduras Transportation project, the Independent Evaluator used two methods: (i) a model-based approach, in which the treatment effect is represented by change in travel time, and the program impact is represented as a function of change in travel time caused by the program intervention. The model relies heavily on geographic information system (GIS) for several purposes, including the estimation of changes in travel times; and (ii) HDM-IV analysis.
The evaluation mode-based approach recognizes that the Honduran road system functions as a single, integrated road network, thereby allowing for network effects to be taken into account. In other words, it takes into account the fact that improvement to a single road section is likely to have impacts that are felt across the entire road network, not just locally, and these impacts may differ depending on where in the road network the improvement section is located, and the degree to which the section serves as a key access point between different sections of the overall network. This new model, which represents the physical road network as an integrated computer/mathematical network (through the GIS), recognizes that in reality, rural households are likely to benefit not only from rural-road improvements, but also from improvements to secondary (or even primary) roads.
|Methodology||Matching with Continuous Treatment Effect|
|Assumptions/ Limitations||These results are subject to some reasonable caveats including: limited generalizability of results; the travel time measures used in the model lacked precision; some potentially interesting direct effects were not studied; and the presence of any unobserved, time-varying variables correlated with travel times could bias these results.|
|Exposure Period||August 2008 (Baseline); March 2011 (Endline). Most roads were completed in 2009-2010|
|Immediate and Intermediate Outcomes||The Ordinary-Least-Squares (OLS) estimate of Average Treatment Effect (ATE) based on continuous treatment variables show that the program intervention had a statistically significant effect (of the expected sign) on many of the access times and costs. For example, cost for travel to hospital decreased 3.53 lempiras ($0.17) and cost to health center decreased 0.194 lempiras ($0.01).|
|Ultimate Impact||The Ordinary-Least-Squares (OLS) estimate of Average Treatment Effect (ATE) based on continuous treatment variables show that the program intervention had a statistically significant effect on increasing monthly agriculture income by 71.9 lempiras ($3.50) and decreasing monthly non-agriculture income by 109 lempiras ($5).|
|Methodology||Highway Development and Management (HDM)-IV|
|Assumptions/ Limitations||Please reference Section IV of the Evaluator Report (page 26-34)|
|Exposure Period||Traffic surveys from 2009, 2010, 2011. Most roads were completed in 2009-2010|
|CA-5 Highway||Despite lower than predicted vehicle traffic counts (which directly impact the calculation of Net Present Value benefits through Vehicle Operating Costs (VOC) as well as ERR estimates), higher than expected road maintenance costs, as well as final project improvement costs which were considerably higher than the previous 2008 estimates used, NORC estimates profitable ERRs (using a 10 percent criterion) for all primary road CA-5 sections except for Section 2 (ERR 7.6 percent), with an ERR range from 12.1 percent - 21.3 percent.|
|Secondary Roads||The ERR estimates for secondary roads in particular were very strong, and reflected a very strong increase in traffic volumes post-project relative to pre-project measures. Further, MCC secondary road project costs were relatively low compared to primary improvement costs, and road improvement designs and work standards were judged by NORC experts to be very high, with very low projected future road deterioration rates due to the high standards of the improvements. Thus, the MCC secondary road improvements appear to have been a very profitable success and these are reflected in their estimated ERR values which ranged from 29.4 percent to 188.3 percent.|
|Rural Roads||It should be noted that there are a number of important caveats regarding using HDM-4 to produce rural road ERR estimates. The HDM-4 rural road ERR estimates are very sensitive to assumptions on future traffic growth and vehicle speeds post improvements: for rural roads, calculated Net Present Values and ERRs tend to correlate closely with traffic volumetric flows. Further, rural road project costs are relatively very low. For these reasons, the economic analysis can in some cases provide very high rates of return due in large part to these low project costs. However, the NORC estimates ERR values ranging from -9.8 percent - 297.7 percent.|
There were several key lessons learned from this evaluation for MCC and future partner countries to consider when designing and implementing roads projects and evaluations:
- Set realistic time horizons. Inevitably there are delays in large infrastructure projects. From the beginning, implementers and evaluators should build into the evaluation design actions for mitigating risk to the evaluation associated with delays in implementation. In the case of Honduras, given the delays in implementation and inflexibility in the evaluation schedule, the exposure period to the improved roads network in some cases was only 5-6 months, whereas for some sections of CA-5 highway rehabilitation wasn’t completed at the time of endline data collection. This is a limited exposure period when decision makers are interested in looking at longer term outcomes, such as changes in prices and income.
- Understand your target beneficiary population. For this evaluation, the target population for the household survey was the population of all households in Honduras at the beginning and end of the project. For the evaluation, the Evaluator used a sample frame constructed for the most recent national census. By focusing on the broader Honduran population, while the evaluation looks at average effects across the country, it is reasonable to expect that some specific groups, particularly those who live closer to the roads upgraded and businesses that rely on the Highway CA-5, would benefit more from the investments.
- Structure evaluation to ask about sustainability. One of the peer reviewers of the Network Model report commented that for any evaluation of road improvement projects, understanding if and why poor road construction outcomes occurred is really important. For future MCC Roads evaluations, we would like to try and answer questions like: Was poor road construction an outcome? If so, then we should explore ways to mitigate this in the future.
As a result of a recent realignment of the Department of Compact Operations, the transport practice is now organized as a single business unit with various staff under one Practice Lead and Senior Director (PLSD) with expertise in transport sector. Work is underway by the transport unit to:
- Develop and put in place project management tools and procedures that will significantly enhance consistency and quality of practices during the development and implementation of transport projects.
- Specific guidelines and templates will be developed to support various aspects of MCC’s transport practices ranging from data collection to the application HDM-IV for planning and decision support.
- Ensure stricter accountability in the planning, design and construction of transport facilities so projects are executed efficiently, cost effectively and in a timely manner.
- Ensure closer monitoring of implementation oversight by the PLSDs as a result of PLSDs given operational authority over the management of their projects in their respective portfolios at each stage of compact development and implementation.
In addition, the PLSD will be the primary POC with Econ and M&E during the planning and implementation of any evaluation of transport projects. Specific transport staff will be assigned by PLSD for specific evaluation exercises as may be required.
These actions, combined with strengthening of the capacity of the MCA staff for improved oversight are expected to reduce construction delays and improve the overall quality of the road projects.
In addition, as a result of these lessons learned, MCC evaluation practices have changed in the following way:
- Institutionalize a formal review process for evaluations: The Monitoring and Evaluation unit is pilot testing a formal review process that defines critical milestones in the evaluation cycle that require substantive review and clearance by key internal stakeholders. This review process also requires local stakeholder review of key evaluation documents in consultation with the evaluator prior to submission to MCC in order to provide feedback on feasibility of proposed evaluation, as well as technical, and factual accuracy of evaluation documents. The formal review process is intended to ensure that evaluations are designed with stakeholder buy-in, are designed using the program logic, use appropriate methodologies for the timeframe of the expected results, and are flexible enough to adjust to changes in implementation.
- Institutionalize evaluation risk assessment: An Evaluation Risk Assessment Checklist has been developed and institutionalized by the Monitoring and Evaluation unit. The risk assessment checklist is reviewed by the M&E lead with M&E management. The risk assessment is intended to inform decision making and identify necessary course correction for more timely response to risk identification.
- Institutionalize development and use of standardized evaluation templates: The Monitoring and Evaluation unit has developed standardized templates in order to provide guidance internally and to independent evaluators on expectations related to evaluation activities and products. These templates are intended to clarify and raise standards for evaluations by influencing the daily work of M&E staff and evaluators.
There are no next steps planned for this evaluation.