Integrating complex programs and rigorous evaluation designs can be challenging but is particularly important to “get right” when implementing pilot programs. The implementer argued that community mobilization required adaptation and believed the evaluation design constrained their ability to mobilize and implement effectively. However, for the very reason that the implementer could not tell ex ante using observable characteristics which communities would participate or succeed, it would have been impossible to use observable information to identify comparison communities and generate the kind of unbiased estimates that a randomized controlled trial (RCT) offers. Unbiased estimates of program impact are critical to assessing whether a program is worth repeating or expanding, which is the entire purpose of conducting a pilot.
Lesson Learned