Income impact is the capstone of the Millennium Challenge Corporation’s “continuum of results” framework. Independent evaluations are the most rigorous means of measuring that impact and at the heart of MCC’s commitment to accountability, learning, transparency, and evidence-based decision making. Independent evaluations, which are conducted by third-party independent experts, help answer three fundamental questions:
- Was MCC’s investment implemented according to plan? This is key to transparency.
- What are the changes in income for program participants that are attributable to MCC’s investment? This is key to accountability.
- Why did or didn’t the planned investments lead to changes in income? This is key to learning
Impact and Performance
Impact evaluations are the most rigorous form of evaluations because they make it possible to know whether the observed impacts were caused specifically by an MCC investment or, alternatively, are the result of external factors that affected program participants and non-participants, like increased market prices for agricultural goods, national policy changes or favorable weather conditions. Impact evaluations compare what happened with the MCC investment to what would have happened without it, through the use of a counterfactual.
Performance evaluations are a valuable tool for estimating the contribution of MCC investments to changes in trends on outcomes, including farm and household income. Performance evaluations are less rigorous and cannot attribute causal impact to MCC investments because they do not use a counterfactual. However, they are useful to compare changes in the situation before and after MCC’s investment and provide details on how an investment might have contributed to changes in beneficiary income, how it might have contributed to changes in outcomes and why.
How We Choose
There are several critical factors that MCC considers when deciding to invest in an impact or a performance evaluation:
- Learning potential: For programs where the assumptions underlying the project logic are based on limited evidence, there is a strong case for an impact evaluation. A rigorous impact evaluation tests assumptions about a project’s effectiveness and contributes substantially to MCC’s future decision-making, as well as the global evidence base.
- Feasibility: The feasibility of designing and implementing a strong impact evaluation is based on how well the evaluators are able to estimate a counterfactual and how feasible it is to maintain that counterfactual through the duration of the evaluation period.
- Strong stakeholder commitment: Identifying a control group and ensuring adherence to an impact evaluation design require significant commitment and collaboration by sector staff, program implementers and evaluators, both within MCC and among partner countries.
- Appropriate timing: The evaluation timeline must be informed by the project logic, particularly with regard to assumptions about how long it will take for expected impacts to occur. By collecting data too early, evaluations may underestimate the impacts on outcomes of interest or miss important lessons.
- Proper coordination: Evaluations require close coordination with program implementation. Program designers, implementers and evaluators must work together to understand and define the program logic, estimate how long expected impacts are likely to take to accrue and identify what is most important to learn about how the program works. This is particularly true for impact evaluations, which require coordination and commitment among various stakeholders to estimate a counterfactual.
Incorporating evaluations—particularly impact evaluations—into program operations is not easy, but this is a challenge that MCC embraces to ensure accountability for results and to improve learning on what works. This commitment to evaluation helps distinguish MCC in the international development community.
An impact evaluation is defined by the ability to estimate the counterfactual, what would have happened to the same group of program participants if they had not received MCC’s assistance. The most rigorous method for estimating the counterfactual to measure attributable program impacts is through randomized control trials. In many programs, there are financial and/or logistical constraints to providing all eligible individuals or groups with an intervention. Random selection (such as through a lottery) is a fair and transparent way to select which eligible individuals or groups should receive the intervention first.
Because randomized control trials randomly select individuals that will and will not be exposed to program benefits, evaluators can compare the groups to measure their impacts. This use of a statistically identical control group creates the greatest opportunity for learning what works and for measuring program impacts.
When a randomized control trial is not feasible, MCC may use other methods to construct a credible comparison group, such as a propensity score matching, difference in differences or regression discontinuity.
List of Evaluations
This table lists planned and completed independent evaluations.
|Country Program||Activity Evaluated||Focus of Evaluation||Type||Status|
|Albania Threshold Program||Albania Thresholds Evaluation||Performance||Planned|
|Albania Threshold Program, Stage II||Albania Thresholds Evaluation||Performance||Planned|
|Armenia Compact||Armenia Compact Water to Market Farmer Training||Impact||Planned|
|Armenia Compact||Armenia Water to Market Institutional Strengthening Performance Evaluation||Performance||Planned|
|Armenia Compact||Armenia Irrigated Agriculture Infrastructure Project Evaluation||
What is the impact of canal improvements on the quality and reliability of irrigation water, agricultural productivity and household income?
|Armenia Compact||Armenia’s Water to Market - Farmer Training Activity Evaluation||
What is the impact of on-farm water management training on farming practices, agricultural productivity, and the income of rural farming households?
|Armenia Compact||Armenia Rural Road Rehabilitation Project Evaluation||
What is the effect of rural road rehabilitation on the quality and accessibility of roads, agricultural productivity and profits, and household well-being?
|Benin Compact||Benin Access to Justice Evaluation||Performance||Planned|
|Benin Compact||Benin’s Access to Land Project Evaluation||
What is the impact of land tenure on investment and income?
|Burkina Faso Compact||Burkina Faso Roads Project Evaluation||Impact||Planned|
|Burkina Faso Compact||Burkina Faso Land Tenure Project Evaluation||Impact||Planned|
|Burkina Faso Compact||Burkina Faso Bright 2 School Project Evaluation||Impact||Planned|
|Burkina Faso Compact||Burkina Faso Agriculture Project Evaluation||Performance||Planned|
|Burkina Faso Threshold Program||Burkina Faso’s Threshold Program Evaluation||
What was the impact of the program on school enrollment? What was the impact of the program on test scores? Were the impacts different for girls than for boys?
|Cabo Verde Compact||Cape Verde Roads and Bridges Activity||Performance||Planned|
|Cabo Verde Compact||Cape Verde Private Sector Development Project Evaluation||Performance||Completed|
|Cabo Verde Compact||Cape Verde Watershed Management & Agricultural Support Project Evaluation||Performance||Planned|
|Cabo Verde Compact II||Cape Verde II Land Management for Investment Project Evaluation||Performance||Planned|
|Cabo Verde Compact II||Cape Verde II Water, Sanitation and Hygiene Project Evaluation||Performance||Planned|
|El Salvador Compact||El Salvador Investment Support Activity Evaluation||Performance||Planned|
|El Salvador Compact||El Salvador Chalatenango Technical Institute (ITCHA) Sub-Activity Evaluation||Performance||Planned|
|El Salvador Compact||El Salvador Non-Formal Training Sub-Activity Evaluation||Performance||Planned|
|El Salvador Compact||El Salvador Scholarships Sub-Activity Evaluation||
What is the impact of secondary school scholarships on recipients’ education and labor market outcomes?
|El Salvador Compact||El Salvador Production and Business Services Activity Evaluation||
What impact did the offer of productive development services have on beneficiaries’
|El Salvador Compact||El Salvador’s Rural Electrification Sub-Activity Evaluation||
What is the impact of electrification on the cost of energy, energy consumption, time allocation, and household income?
|El Salvador Compact||El Salvador’s Technical Middle Schools Sub-Activity Evaluation||
What is the impact of improved technical middle schools on completion rates, employment, and income?
|El Salvador Compact||El Salvador Connectivity Project Evaluation||
What is the impact of road improvements on travel cost and time, land prices, and household income?
|El Salvador Compact||El Salvador Water and Sanitation Sub-Activity Evaluation||
What is the impact of water and sanitation on the cost of water, water consumption, illness, time use, and household income?
|Georgia Compact||Georgia Samtskhe-Javakheti Road Rehabilitation Activity||
How does the road rehabilitation effect/cause economic development, new businesses, and economic and social integration in the region?
|Georgia Compact||Georgia Agribusiness Development Activity (ADA) Evaluation||
How does the provision of ADA grants to farmers and farm-related businesses impact household income, poverty levels, and job creation?
|Ghana Compact||Ghana Trunk Roads Activity Evaluation||Performance||Planned|
|Ghana Compact||Ghana National Highway Activity Evaluation||Performance||Planned|
|Ghana Compact||Ghana Ferry Activity Evaluation||Performance||Planned|
|Ghana Compact||Ghana Credit Activity Evaluation||Performance||Planned|
|Ghana Compact||Ghana Post-Harvest Activity Evaluation||Performance||Planned|
|Ghana Compact||Ghana Irrigation Activity Evaluation||Performance||Planned|
|Ghana Compact||Ghana Land Activity Evaluation||Impact||Planned|
|Ghana Compact||Ghana Water and Sanitation Sub-Activity Evaluation||
How has the investment in water systems improved health conditions for the beneficiary communities? Are the improvements in health conditions attributable to the improved water systems, and why? Have the improvements in health resulted in increased availability of labor for economic activities?
|Ghana Compact||Ghana’s Feeder Roads Activity Evaluation||
Do improved roads lead to higher farm income through reduced input cost and higher producer price at the farm gate that are associated with reduced travel time and vehicle operating cost (transport cost)?
|Ghana Compact||Ghana Commercial Training Activity Evaluation||
Does the FBO training program cause farmers to adopt new technologies or techniques, such as using land more intensively and efficiently, choosing crops that are more competitive, or optimizing the use of inputs, including labor? What is the magnitude of any spillover from the trained farmers on proximate farmers and those in the trained farmers’ social networks? Does the FBO training program cause farmers to increase their yields, sales, incomes, and enhance their access to social services?
|Honduras Compact||Honduras Farmer Access to Credit Activity Evaluation||Performance||Planned|
|Honduras Compact||Honduras’ Transportation Project Evaluation||
Do decreased transport costs lead to increased incomes?
|Honduras Compact||Honduras’ Farmer Training and Development Activity Evaluation||
What is the impact of increased productivity and business skills on incomes?
|Indonesia Threshold Program||Indonesia Threshold Evaluation||Performance||Completed|
|Lesotho Compact||Lesotho Health Sector Project Evaluation||Performance||Planned|
|Lesotho Compact||Lesotho Urban and Peri-Urban Water Activity Evaluation||Performance||Planned|
|Lesotho Compact||Lesotho Rural Water Supply and Sanitation Activity Evaluation||Impact||Planned|
|Lesotho Compact||Lesotho Land Administration Reform Activity Evaluation||Impact||Planned|
|Malawi Threshold Program||Malawi Threshold Evaluation||Performance||Planned|
|Mali Compact||Mali Bamako-Senou Airport Improvement Project Evaluation||Performance||Planned|
|Mali Compact||Mali Alatona Irrigation Project Evaluation||Impact||Planned|
|Moldova Compact||Moldova Grow HVA Sales Activity Evaluation||Impact||Planned|
|Moldova Compact||Moldova Irrigation Sector Reform and Rehab of CIS Activities Evaluation||Impact||Planned|
|Moldova Compact||Moldova Road Rehabilitation Project Evaluation||Performance||Planned|
|Mongolia Compact||Mongolia Vocational Education Project||Impact||Planned|
|Mongolia Compact||Mongolia Peri-Urban Land Leasing Activity Evaluation||Impact||Planned|
|Mongolia Compact||Mongolia Privatization of Ger Area Land Plots and Land Registration Activities Evaluation||Impact||Planned|
|Mongolia Compact||Mongolia Health Project Evaluation||Performance||Planned|
|Mongolia Compact||Mongolia Energy and Environmental Project||Impact||Planned|
|Morocco Compact||Morocco Small-Scale Fisheries Project Evaluation||Performance||Planned|
|Morocco Compact||Morocco Rain-Fed Olive, Almond & Fig Tree Expansion Sub-Activity Evaluation||Performance||Planned|
|Morocco Compact||Morocco Rain-Fed Olive, Almond & Fig Tree Rehabilitation Sub-Activity Evaluation||Impact||Planned|
|Morocco Compact||Morocco Tree Irrigation and Intensification Activities Evaluation||Performance||Planned|
|Morocco Compact||Morocco Financial Services Project Evaluation||Performance||Planned|
|Morocco Compact||Morocco Enterprise Support Project Evaluation||Impact||Planned|
|Morocco Compact||Morocco Artisan and Fez Medina Project Evaluation||Performance||Planned|
|Mozambique Compact||Mozambique Municipal Sanitation and Drainage Systems Activity Evaluation||Performance||Planned|
|Mozambique Compact||Mozambique Land Tenure Project Evaluation||Impact||Planned|
|Mozambique Compact||Mozambique Farmer Income Support Project Evaluation||Performance||Planned|
|Mozambique Compact||Mozambique’s Rural Water Supply Project Evaluation||
Does increased access to improved rural water points improve health outcomes with respect to diarrhea in particular and reduce time to collect water? Do improved and appropriate technology of rural water points increase household income and raise productivity?
|Namibia Compact||Namibia Vocational and Skills Training Activity||Impact||Planned|
|Namibia Compact||Namibia Regional Study and Resource Centers Activity Evaluation||Performance||Planned|
|Namibia Compact||Namibia Livestock Market Efficiency Fund Sub-Activity Evaluation||Performance||Planned|
|Namibia Compact||Namibia Community-Based Rangeland & Livestock Management Sub-Activity Evaluation||Impact||Planned|
|Namibia Compact||Namibia Indigenous Natural Products and Ecotourism Development Activities Evaluation||Performance||Planned|
|Nicaragua Compact||Nicaragua Rice Sub-Activity Evaluation||Performance||Planned|
|Nicaragua Compact||Nicaragua Plantain Sub-Activity Evaluation||Performance||Planned|
|Nicaragua Compact||Nicaragua ProNicaragua Sub-Activity Evaluation||Performance||Planned|
|Nicaragua Compact||Nicaragua Farming and Forestry Activity Evaluation||Performance||Planned|
|Nicaragua Compact||Nicaragua Property Regulation Project Evaluation||Performance||Planned|
|Nicaragua Compact||Nicaragua Transportation Project Evaluation||
Does a reduction in transport costs and travel times lead to increased movement of goods and people on roads and increased access to goods and decreased costs of consumption, services and inputs? Was the project cost effective, as analyzed through re-estimated economic rates of return, comparisons to original estimates, and assessment of impact?
|Nicaragua Compact||Nicaragua’s Rural Business Development Services Activity Evaluation||
What is the impact of agriculture technical assistance on the well-being (consumption) of beneficiaries?
|Niger Threshold Program||Niger Threshold Program II Evaluation||Impact||Planned|
|Niger Threshold Program||Niger Threshold Program I Evaluation||
Do the program interventions, independently or in combination, result in increases in girls’ primary education enrollment, attendance and completion rates?
|Philippines Compact||Philippines Roads Project Evaluation||Performance||Planned|
|Philippines Compact||Philippines Revenue Administration Reform Project Evaluation||Performance||Planned|
|Philippines Compact||Philippines Community Development Grants Project Evaluation||Impact||Completed|
|Rwanda Threshold Program||Rwanda Threshold - Strengthening Civic Participation Evaluation||Impact||Planned|
|Senegal Compact||Senegal Roads Project||Impact||Planned|
|Senegal Compact||Senegal Irrigation Project Evaluation||Impact||Planned|
|Tanzania Compact||Tanzania Morogoro Water Supply Activity Evaluation||Impact||Planned|
|Tanzania Compact||Tanzania Lower Ruvu Plant Expansion Activity Evaluation||Impact||Planned|
|Tanzania Compact||Tanzania Kigoma Solar Power Activity Evaluation||Performance||Planned|
|Tanzania Compact||Tanzania Zanzibar Interconnector Activity Evaluation||Performance||Planned|
|Tanzania Compact||Tanzania Customer Connection Financing Program Sub-Activity Evaluation||Impact||Planned|
|Tanzania Compact||Tanzania’s Mafia Island Airport Activity Evaluation||Performance||Planned|
|Tanzania Compact||Tanzania Mainland Trunk Roads & Zanzibar Rural Roads Evaluations||
Does a reduction in transport costs and travel times lead to increased access to markets and economic activity in towns/villages near a road?
|Tanzania Compact||Tanzania Electricity Distribution Systems Rehabilitation & Extension Activity Evaluation||
Does access to electricity lead to a) increased household income and better health and education outcomes and b) increased business activity, including new firms, capital investments and greater levels of investment? If impact is detected, what is the magnitude of this impact and what is the magnitude in comparison to the costs?
|Tanzania Threshold Program||Tanzania Threshold Evaluation||Performance||Completed|
|Vanuatu Compact||Vanuatu Transport Infrastructure Project Evaluation||Performance||Planned|