Ensure sufficient statistical power. The evaluation included a small number of road sections – 27 in the treatment group and 29 in the comparison group. Since there is intra-cluster correlation amongst households located around any one road section, the statistical power of the evaluation was limited. As a result, it is challenging to interpret the results for the medium and long term outcomes. Was there really very little impact or were the impacts just not as large as the minimum detectable effects, which for many medium and long term variables were quite high. The weak statistical power was due to two main factors: (1) a small number of road sections actually being rehabilitated and (2) design of the evaluation around an existing data source which limited the number of road sections covered and the sample size. In the future, before investing significant resources into an evaluation, MCC should ensure that there will be sufficient statistical power for measuring realistic changes in key outcomes. To achieve this goal, it may be productive to for an evaluation to cover similar programs in multiple countries rather than for each evaluation to only examine one country at a time.
Lesson Learned