The randomized roll-out evaluation approach has risks. For the farmer training impact evaluation, the evaluators used a randomized roll-out approach in which a first round of treatment farmers is compared to a control group of farmers that received training at a later date. The key to this approach is that there is enough time between the two phases to see behavior change and the accrual of benefits for the first farmers before the second round of farmers is trained. Timelines for farmer adoption of new practices, the five-year compact timeline and inevitable implementation delays made the randomized roll-out a risky approach. In the case of Armenia, the timing was such that the on-farm water management and high value agriculture control group was trained before the Irrigation Infrastructure Activity was completed, thereby losing the ability to compare between the two groups once irrigation was in place. Given the loss of the counterfactual, it is not possible to estimate the causal impact of the training on outcomes with the completed irrigation infrastructure or even to allow for more crop cycles and an adjusted (more realistic) timeline for behavior change. This is a potential risk that should be considered for future impact evaluations using a randomized roll-out methodology.
Lesson Learned