Independent Evaluations

Income impact is the capstone of the Millennium Challenge Corporation’s “continuum of results” framework. Independent evaluations are the most rigorous means of measuring that impact and at the heart of MCC’s commitment to accountability, learning, transparency, and evidence-based decision making. Independent evaluations, which are conducted by third-party independent experts, help answer three fundamental questions:

Was MCC’s investment implemented according to plan? This is key to transparency.
What are the changes in income for program participants that are attributable to MCC’s investment? This is key to accountability.
Why did or didn’t the planned investments lead to changes in income? This is key to learning

Impact and Performance

Impact Evaluations

Impact evaluations are the most rigorous form of evaluations because they make it possible to know whether the observed impacts were caused specifically by an MCC investment or, alternatively, are the result of external factors that affected program participants and non-participants, like increased market prices for agricultural goods, national policy changes or favorable weather conditions. Impact evaluations compare what happened with the MCC investment to what would have happened without it, through the use of a counterfactual.

Performance Evaluations

Performance evaluations are a valuable tool for estimating the contribution of MCC investments to changes in trends on outcomes, including farm and household income. Performance evaluations are less rigorous and cannot attribute causal impact to MCC investments because they do not use a counterfactual. However, they are useful to compare changes in the situation before and after MCC’s investment and provide details on how an investment might have contributed to changes in beneficiary income, how it might have contributed to changes in outcomes and why.

How We Choose

There are several critical factors that MCC considers when deciding to invest in an impact or a performance evaluation:

Learning potential: For programs where the assumptions underlying the project logic are based on limited evidence, there is a strong case for an impact evaluation. A rigorous impact evaluation tests assumptions about a project’s effectiveness and contributes substantially to MCC’s future decision-making, as well as the global evidence base.
Feasibility: The feasibility of designing and implementing a strong impact evaluation is based on how well the evaluators are able to estimate a counterfactual and how feasible it is to maintain that counterfactual through the duration of the evaluation period.
Strong stakeholder commitment: Identifying a control group and ensuring adherence to an impact evaluation design require significant commitment and collaboration by sector staff, program implementers and evaluators, both within MCC and among partner countries.
Appropriate timing: The evaluation timeline must be informed by the project logic, particularly with regard to assumptions about how long it will take for expected impacts to occur. By collecting data too early, evaluations may underestimate the impacts on outcomes of interest or miss important lessons.
Proper coordination: Evaluations require close coordination with program implementation. Program designers, implementers and evaluators must work together to understand and define the program logic, estimate how long expected impacts are likely to take to accrue and identify what is most important to learn about how the program works. This is particularly true for impact evaluations, which require coordination and commitment among various stakeholders to estimate a counterfactual.

Incorporating evaluations—particularly impact evaluations—into program operations is not easy, but this is a challenge that MCC embraces to ensure accountability for results and to improve learning on what works. This commitment to evaluation helps distinguish MCC in the international development community.

Download the fact sheet

The Counterfactual

An impact evaluation is defined by the ability to estimate the counterfactual, what would have happened to the same group of program participants if they had not received MCC’s assistance. The most rigorous method for estimating the counterfactual to measure attributable program impacts is through randomized control trials. In many programs, there are financial and/or logistical constraints to providing all eligible individuals or groups with an intervention. Random selection (such as through a lottery) is a fair and transparent way to select which eligible individuals or groups should receive the intervention first.

Because randomized control trials randomly select individuals that will and will not be exposed to program benefits, evaluators can compare the groups to measure their impacts. This use of a statistically identical control group creates the greatest opportunity for learning what works and for measuring program impacts.

When a randomized control trial is not feasible, MCC may use other methods to construct a credible comparison group, such as a propensity score matching, difference in differences or regression discontinuity.

List of Evaluations

This table lists planned and completed independent evaluations.

Independent Evaluations
Country Program	Activity Evaluated	Focus of Evaluation	Type	Status
Armenia Compact	Impact Evaluation for Armenia’s Rural Road Rehabilitation	What is the effect of rural road rehabilitation on the quality and accessibility of roads, agricultural productivity and profits, and household well-being?	Impact	Planned
Benin Compact	Impact Evaluation for Benin’s Access to Land Project	What is the impact of land tenure on investment and income?	Impact	Planned
Benin Compact	Impact Evaluation for Benin’s Financial Innovation and Expansion Challenge Facility	What is the impact of the intervention on transaction costs and loan portfolio quality, and deposit and lending activity of microfinance institutions? What is the impact on the operating costs and profits of grantees? Does the intervention improve the financial performance of clients? (cash flow, gain better access to credit)	Impact	Planned
El Salvador Compact	Impact Evaluation for El Salvador’s Water and Sanitation Sub-Activity	What is the impact of water and sanitation on the cost of water, water consumption, illness, time use, and household income?	Impact	Planned
Armenia Compact	Impact Evaluation for Armenia’s Water to Market Activity	What is the impact of on-farm water management training on farming practices, agricultural productivity, and the income of rural farming households?	Impact	Completed
Burkina Faso Threshold Program	Impact Evaluation for Burkina Faso’s Threshold Program	What was the impact of the program on school enrollment? What was the impact of the program on test scores? Were the impacts different for girls than for boys?	Impact	Completed
El Salvador Compact	Impact Evaluation for El Salvador’s Connectivity Project	What is the impact of road improvements on travel cost and time, land prices, and household income?	Impact	Planned
El Salvador Compact	Impact Evaluation for El Salvador’s Education and Training Activity	What is the impact of improved technical middle schools on completion rates, employment, and income?	Impact	Planned
El Salvador Compact	Impact Evaluation for El Salvador’s Rural Electrification Sub-Activity	What is the impact of electrification on the cost of energy, energy consumption, time allocation, and household income?	Impact	Completed
Georgia Compact	Impact Evaluation for Georgia’s Agribusiness Development Activity (ADA)	How does the provision of ADA grants to farmers and farm-related businesses impact household income, poverty levels, and job creation?	Performance	Planned
Georgia Compact	Impact Evaluation for Georgia’s Samtskhe-Javakheti Road Rehabilitation Activity	How does the road rehabilitation effect/cause economic development, new businesses, and economic and social integration in the region?	Impact	Planned
Ghana Compact	Impact Evaluation for Commercial Training in Ghana	Does the FBO training program cause farmers to adopt new technologies or techniques, such as using land more intensively and efficiently, choosing crops that are more competitive, or optimizing the use of inputs, including labor? What is the magnitude of any spillover from the trained farmers on proximate farmers and those in the trained farmers’ social networks? Does the FBO training program cause farmers to increase their yields, sales, incomes, and enhance their access to social services?	Impact	Completed
Ghana Compact	Impact Evaluation for Ghana’s Feeder Roads Activity	Do improved roads lead to higher farm income through reduced input cost and higher producer price at the farm gate that are associated with reduced travel time and vehicle operating cost (transport cost)?	Impact	Planned
Honduras Compact	Independent Evaluation for Honduras’ Farmer Training and Development Activity	What is the impact of increased productivity and business skills on incomes?	Impact	Completed
Honduras Compact	Impact Evaluation for Honduras’ Transportation Project	Do decreased transport costs lead to increased incomes?	Impact	Completed
Mozambique Compact	Impact Evaluation for Mozambique’s Rural Water Supply Project	Does increased access to improved rural water points improve health outcomes with respect to diarrhea in particular and reduce time to collect water? Do improved and appropriate technology of rural water points increase household income and raise productivity?	Impact	Planned
Niger Threshold Program	Impact Evaluation for Niger’s Threshold Program	Do the program interventions, independently or in combination, result in increases in girls’ primary education enrollment, attendance and completion rates?	Impact	Planned
Tanzania Compact	Impact Evaluation for Tanzania Electricity Distribution Systems Rehabilitation & Extension Activity	Does access to electricity lead to a) increased household income and better health and education outcomes and b) increased business activity, including new firms, capital investments and greater levels of investment? If impact is detected, what is the magnitude of this impact and what is the magnitude in comparison to the costs?	Impact	Planned
Tanzania Compact	Impact Evaluation for Tanzania’s Mainland Trunk Roads & Zanzibar Rural Roads	Does a reduction in transport costs and travel times lead to increased access to markets and economic activity in towns/villages near a road?	Impact	Planned
Nicaragua Compact	Impact Evaluation for Nicaragua’s Rural Business Development Services Activity	What is the impact of agriculture technical assistance on the well-being (consumption) of beneficiaries?	Impact	Completed
Armenia Compact	Impact Evaluation for Irrigation Infrastructure Improvements in Armenia	What is the impact of canal improvements on the quality and reliability of irrigation water, agricultural productivity and household income?	Impact	Planned
El Salvador Compact	Impact Evaluation for the Production and Business Services Activity in El Salvador	What impact did the offer of productive development services have on beneficiaries’ incomes and employment?	Impact	Completed
El Salvador Compact	Impact Evaluation for Scholarships in El Salvador	What is the impact of secondary school scholarships on recipients’ education and labor market outcomes?	Impact	Planned
Nicaragua Compact	Impact Evaluation for Road Improvements in Nicaragua	Does a reduction in transport costs and travel times lead to increased movement of goods and people on roads and increased access to goods and decreased costs of consumption, services and inputs? Was the project cost effective, as analyzed through re-estimated economic rates of return, comparisons to original estimates, and assessment of impact?	Impact	Completed
Ghana Compact	Impact Evaluation for Community Services Water Activity in Ghana	How has the investment in water systems improved health conditions for the beneficiary communities? Are the improvements in health conditions attributable to the improved water systems, and why? Have the improvements in health resulted in increased availability of labor for economic activities?	Impact	Planned
Tanzania Compact	Performance Evaluation for Tanzania’s Mafia Island Airport Activity		Performance	Planned