Before proceeding with an impact evaluation endline and analysis, MCC should carefully consider the risks, constraints, and compromises to the evaluation design, and the resulting implications for the expected learning value of the evaluation. The impact evaluation design was compromised in important ways by a number of factors including i) incomplete take-up, as many of the intended beneficiaries who were expected to have received land use property rights certificates (DUATs) had not, ii) high attrition (31%) between baseline and follow-up, and iii) contamination of parts of the control group, as subsequent projects issued DUATs to some of the control households, as well as significant differences between treatment and control groups at baseline. These issues substantially reduce the statistical power of the analysis, and pose challenges for inferring causality from the matched difference-in-difference analysis. In addition, the long exposure period of the evaluation - over 10 years after implementation of some activities - seems likely to have limited its ability to measure shorter-term impacts, such as increased knowledge and awareness resulting from the project’s outreach campaign. As a result, the findings from the impact evaluation analysis must be interpreted with a high degree of caution, and their implications for future program design are limited.
Lesson Learned