Measuring Results of the Burkina Faso BRIGHT II Project

The MCC compact with Burkina Faso was a five-year investment (2009-2014) of $480.5 million. The $26.8 million BRIGHT II component is the subject of an independent impact evaluation summarized here.

The BRIGHT II Project built classrooms (396 in all) for grades 3-6 for each of the 132 primary schools constructed under the Threshold BRIGHT I Project, with the ultimate goal of increasing girls’ enrollment and primary school completion rates
The BRIGHT and BRIGHT II projects had statistically significant impacts on: self-reported enrollment, overall test scores, and primary school completion rates.  An effect was also detected for the likelihood that young adults (13-22) were employed rather than still in school and girls (though not boys) were less likely to be married.
The evaluation suggests that higher test scores were driven by improved access to girl-friendly schools, but that efforts to address weak overall performance of schools are still necessary to improve the rate of learning.
This Final Evaluation Report represents the last step in this evaluation.

In Context

The MCC compact with Burkina Faso was a five-year investment (2009-2014) of $480.5 million in 4 projects:  the Agriculture Development Project, the Rural Land Governance Project, the Roads Project, and the Burkinabé Response to Improve Girls’ Chances to Succeed (BRIGHT II) Project. The BRIGHT II Project included nine components:  School Construction, Kindergarten (Bisongo) Construction, School Canteens and Take Home Rations, Social Mobilization Campaign, Gender Sensitivity Training, Model Women’s Program, Incentives for Female Teachers, Mother Teacher Associations, and a Literacy Campaign. The $26.8 million BRIGHT II Project is the subject of an independent impact evaluation released by MCC in December 2016, the results of which are summarized here.  This project represents 5.6 percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.

Pie Chart: Distribution of Funding in Burkina Faso I Compact

Program Logic

The BRIGHT II Project of the Burkina Faso Compact was an extension of the BRIGHT Project that was implemented under an MCC Threshold Program from 2005-2008.  The original BRIGHT project built new primary schools of three classrooms each (one each for grades 1-3) in provinces where girls’ enrollment and educational attainment were lowest.  The BRIGHT II Project built an additional 3 classrooms (for grades 4-6) in each school as well as built latrines, kindergarten facilities (bisongos), and complementary programming such as school canteens and take home rations, literacy training, and gender sensitivity training (extensions of activities that had already begun under BRIGHT I).  Both BRIGHT and BRIGHT II were designed to address the low levels of girls’ participation and achievement in primary school. The focus of BRIGHT/BRIGHT II was on increasing access to education and improving educational facilities.  It did not focus substantially on improving the quality of instruction.

 

That girls’ primary school low enrollment and completions rates were primarily caused by problems of access and community engagementThere were several key assumptions underlying the BRIGHT II program logic during the design of the investment:

  • That increased enrollment would improve overall education outcomes (such as graduation levels and test scores)
 
Visualization of Program Logic

For a more detailed version of the program logic, please refer to page 8 of the Burkina Faso M&E Plan, which can be found here.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

 

Indicators

Level

Baseline (2009)

Actual Achieved (07/2014)

Target

Percent Complete

Number of Additional Classrooms Constructed

Output

0

396

396

100%

Number of Bisongos (Kindergartens) Constructed

Output

0

122

122

100%

Number of Latrines Constructed

Output

0

264

264

100%

Number of Students Enrolled in BRIGHT Schools

Outcome

20,465

22,366

38,619

10%

Number of Boys/Girls Enrolled in Bisongos (Kindergartens)

Outcome

0

8,699

9,440

92%

Percentage of Girls Regularly Attending BRIGHT Schools

Outcome

94%

96%

97%

67%

Girls Promotion Rates to Next Grade in BRIGHT Schools

Outcome

N/A

95%

90%

106%

Boys Promotion Rates to Next Grade in BRIGHT Schools

Outcome

N/A

94%

90%

105%

Number of Girls Graduating from BRIGHT II Schools

Outcome

N/A

2,032

4,301

47%

Number of Boys Graduating from BRIGHT II Schools

Outcome

N/A

2,003

3,783

53%

Source: Closeout ITT from November 2014, which includes data through the end of the compact, based on reporting from USAID (the primary project implementer)

 

The average completion rate of the output targets is 119 percent and targets were met or exceeded in 13 of the 19 output indicators. 1   The average completion rate of outcome targets is 66 percent and targets were met or exceeded in in 7 of the 18 outcome indicators percent and outcome targets is 93 percent; and for 17 of the 35 output and outcome indicators, targets were met or exceeded. 2

 

Evaluation Questions 3

The evaluation was designed to answer questions such as:

  • What was the impact of the program on school enrollment?
  • What was the impact of the program on test scores?
  • What was the impact of the program on child labor and young adult outcomes?
  • Were the impacts different for girls?

 

More detail can be found in the Evaluation Design Report here.

 

Evaluation Results

 

The original BRIGHT Project under the Threshold Program was evaluated by Mathematica Policy Research using a regression discontinuity design.  The final report (2009) from the BRIGHT evaluation is available on MCC’s Evaluation Catalog.  Mathematica Policy Research was later selected to perform an evaluation of the BRIGHT II project.  Given that BRIGHT II was an extension of BRIGHT, using the same schools, the regression discontinuity methodology was maintained for BRIGHT II.  Because BRIGHT II was implemented in all of the same BRIGHT schools, however, it was not possible to disentangle the effects of BRIGHT from the effects of BRIGHT II.  Thus, in effect, the present BRIGHT evaluation estimates the impacts of both BRIGHT and BRIGHT II.  An interim Evaluation Report representing the findings from the first 7 years of exposure to the BRIGHT and BRIGHT II Projects (2005-2012) has been published on MCC’s Evaluation Catalog.  The subject of this Summary of Findings represents the Final Evaluation Report, which covers a 10-year period of exposure to BRIGHT and BRIGHT II (2005-2015).

 

 
Evaluator             Mathematica Policy Research (MPR)
Impact or Performance? Impact
Methodology    Regression Discontinuity
Evaluation Period 2005-2015
Outcomes Enrollment:

  • The evaluation found that the project had a statistically significant and positive impact on self-reported enrollment.
  • In BRIGHT communities, school enrollment was 6 percentage points higher than in non-BRIGHT communities.
  • The effect for girls was even higher, where there was a 10.3 percentage point difference between BRIGHT communities and comparison communities.

 

Test Scores:

  • The evaluation found that the project had a .19 standard deviation positive impact on test scores when comparing children in BRIGHT communities to children in non-BRIGHT communities.
  • Girls appeared to benefit more than boys, with girls scoring .08 standard deviations higher than boys on the tests.
  • Analyses suggest that test scores for BRIGHT and comparison children who achieved the same level of education were very similar, suggesting that the improved test scores were largely the result of children in BRIGHT communities achieving more years of schooling, rather than BRIGHT students testing higher than their comparison school peers.  In concrete terms as an example, BRIGHT 3rd graders are not achieving higher scores than comparison 3rd graders; rather, more BRIGHT children are reaching 3rd grade than children in comparison communities, thereby lifting overall village test scores as compared with villages without BRIGHT schools.

 

[Below are additional outcomes not part of program’s intended outcomes]

Child Labor :

  • Though the interim evaluation found that children in BRIGHT communities were engaging in fewer household chores than in comparison areas, the final evaluation does not find any statistically significant difference between BRIGHT and comparison communities.

 

Young Adult Outcomes:

  • The project had a statistically significant impact on whether young adults (13-22) were currently employed, with 5.5% fewer girls and 5.6% fewer boys employed rather than being in school.
  • Girls in BRIGHT communities were less likely to be married- a difference of 6.3% (there was no marriage effect for boys, however, as boys in rural Burkina generally marry later than girls).
Objective-level Outcomes Primary School Completion Rate:

  • Children in BRIGHT communities were substantially more likely to complete primary school.  Primary school completion rates were 13.5 percent higher in BRIGHT communities.

 

Effect on household income attributable to MCC Household income was not directly measured through the evaluation.

 

Lessons Learned

  • Improvements in enrollment and progression drove improved levels of learning, but without efforts to address weak overall performance of schools, the rate of learning may not improve.  As education access (availability, distance) and girls’ participation were constrained in rural Burkina Faso, the project’s approach of bundled interventions to address these two constraints seem to have driven improvements on these dimensions.  Teacher housing and performance awards seems to have helped BRIGHT schools attract more female teachers and allow schools to reduce teacher/student ratio.  However, despite improving girls’ engagement and smaller classroom sizes, BRIGHT schools did not seem to have a steeper learning profile than non-BRIGHT schools (i.e. how long it takes a given student to arrive at a skill/competency such as recognizing letters or reading a sentence).  In contexts where education quality lags acts as a more binding constraint than enrollment, improvements to the academic environment – including quality instruction, better educational management and leadership, and academic support for struggling students – might help steepen learning profiles.
  • Girls may require specific interventions to improve their educational outcomes and to reduce the likelihood of early marriage.  The BRIGHT evaluation links to other literature suggesting that responsibilities for younger children’s care, cultural and religious attitudes, mothers’ own educational levels, food insecurity and school availability are a mix of factors that negatively impact girls’ schooling.  Low participation in schooling also seems to be linked to early marriage.  Moreover, BRIGHT girls’ educational levels, learning, and likelihood of avoiding early marriage improved at a higher rate than for eligible boys, indicating BRIGHT’s specific offerings motivated families to value girls’ schooling and success in particular.  Understanding the unique trajectories of and constraints facing girls and boys is a necessary prerequisite to identifying impactful solutions.
  • Early consideration should be given regarding the sustainability of any activities that will require institutional support in the longer-term.  The physical structures remain in good condition in BRIGHT villages, but the size of educational impacts have decreased over time, indicating that the “soft interventions” which ended at the conclusion of the BRIGHT II program collectively had an impact on outcomes while they lasted.   While not all supplementary interventions need to continue perpetually after the end of an intervention such as BRIGHT, working towards support/funding strategies with education authorities may ensure continued impacts after the end of donor funding.
  • Project and Evaluation Design Must Match Desired Learning: Because the BRIGHT and BRIGHT II interventions were implemented in all of the same communities, and because all treatment sites received all of the interventions, it was not possible to disaggregate the impacts of different project components.  For instance, it was not possible to know how much of the improvements in enrollment and/or test scores were due to school construction vs. having access to gender-specific latrines vs. activities to increase community engagement in education, etc.  While disaggregating impacts of bundled interventions is always difficult, early attention to matching project and evaluation design to desired learning can help mitigate these issues.  Also, early attention to desired learning can also suggest additional methods (such as qualitative research methods) to address desired research questions that can’t be well answered through quantitative analyses.

 

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. In this case, an indicator which met 99% of the target is being considered met.
  • 2. These figures are calculated using all non-evaluation indicators with targets in the BRIGHT II Project.
  • 3. MCC did not conduct an economic analysis of this project. Therefore it is not possible to compare these evaluation questions to the expected economic benefit streams.