Lesson Learned

Evaluations of complex reform projects should target a concise set of evaluation questions and focus evaluation reports on the sub-set of questions most pertinent at the time of data collection.

Evaluations of complex reform projects should target a concise set of evaluation questions and focus evaluation reports on the sub-set of questions most pertinent at the time of data collection. The Power Sector Reform Project (PSRP) evaluation was designed to answer 23 evaluation questions and it produced three evaluation reports: a process evaluation after the first year of program implementation, interim process evaluation and process mapping results after 3 years of program exposure, and a final process mapping after 5 years of exposure. Instead of having each report attempt to address the full set of evaluation questions at each stage, it may be beneficial to target each report that presents evaluation results on a sub-set of evaluation questions that are most relevant to the time at which data collection occurs. The evaluation requirements for PSRP called for the use of process evaluation methods in addition to stipulating the need to assess key outcomes at the regulatory, institutional and policy level within the power sector. Given MCC’s interest in better understanding the effectiveness of various modalities for implementing reform programs, the use of process evaluation in particular yielded valuable insights that will inform the design of future compacts. However, the overall performance evaluation included a high number of evaluation questions, several of which focused on a narrow set of activities that were less central to the overall logic underlying the theory of change. For some evaluation questions, the findings echoed results reported from other activities but added little additional insight. While MCC and Malawian stakeholders derived significant learning from the varied findings and analyses, at times the length and number of individual reports was found to be overwhelming. Therefore, future MCC should collaborate with evaluators of future reform projects to carefully devise a set of evaluation questions focused on the core objectives of the project while also offering an improved understanding of implementation.