Evaluations of individual grants from a grant facility present unique evaluability challenges that should be contemplated before embarking on grant facilities in future MCC programming:
It is unclear how to apply the concept of accountability for results to a grant facility. In a facility with multiple programmatically diverse and geographically varied grants and when grants are identified midway through a compact and begin implementing shorting after approval, it is generally impossible to conduct a thorough and rigorous evaluation of each funded grant. Given this, MCC should identify what the agency believes grant facility programs should be held accountable for.
Individual grants are not held to the same documentation and due diligence standards as a normal MCC project and therefore rarely have program logics and clear targets that are required to facilitate evaluation. The peatland portfolio of grants was not defined during compact development so evaluation design had to occur after the grants were selected. MCC is working on a leveraged grant facility guidance document that will help address these concerns and move grant identification earlier in the compact development process.
Monitoring multiple de-centralized small grants is difficult, which means data quality varies from grant to grant. This presents challenges as the evaluator has to validate what actually happened instead of getting this information from the indicator tracking table or MCA documents. The evaluation ends up focusing on telling MCC what actually happened during the compact.
MCC is addressing this lesson by considering alternate evaluation approaches to these types of programs. These approaches might include assessing the overarching results of a grant facility, at the objective level, rather than attempting to evaluate a sample of grants in a more detailed manner; or conducting retrospective evaluations of grant facilities and associated grants, which would generally not allow for impact evaluation and may have limited baseline data with which to conduct a pre-post analysis.