Sequencing rounds of data collection and analysis can help ensure efficient use of evaluation resources. Findings from the evaluation’s inception mission suggested variable completion and functionality of the compact-funded infrastructure investments, which prompted considerations about whether different components might warrant different evaluation approaches or any evaluation at all. MCC and the evaluator agreed to sequence phases of the evaluation such that implementation fidelity could be assessed before deciding on the final scope and nature of a summative evaluation that would measure customer-level outcomes. The evaluator and MCC also agreed to sequence qualitative data collection ahead of quantitative data collection to investigate the validity of potential counterfactual groups before committing to a large-scale data collection effort that would have greater costs and respondent burden than other alternatives. (Additionally, some evaluations sequence qualitative data collection to follow quantitative data collection in order to help explore and explain the quantitative findings.) This sequenced approach allowed the evaluator to understand whether the interventions that had been designed actually came to fruition before collecting data on the customer-level outcomes that were expected to result from these interventions. It also allowed them to prioritize the most suitable areas for an impact evaluation design, while ensuring that the other sites were included in the evaluation for accountability and learning purposes. Similarly-sequenced approaches could be helpful in other cases where the evaluator and MCC have uncertainty about the context or appropriateness of design options.
Lesson Learned