Measuring Results of the Niger IMAGINE and NECS Projects

The MCC threshold program with Niger was planned as a three-year investment (March 2008 – September 2011) of $24.7 million. The IMAGINE project and the subsequent NECS project are the subject of an independent impact evaluation summarized here.

The IMAGINE project, as originally designed, consisted of the construction of primary schools and a set of complementary interventions designed to increase girls’ enrollment and completion rates. The NECS project continued and complemented girls’ education activities begun under IMAGINE and also included investments designed to increase student reading achievement.

The NECS project alone had a 9.5 percentage point positive impact on primary school enrollment, an

  • percentage point positive impact on attendance, a 15 standard deviation positive impact on local- language test scores, and no impact on French-language test scores. Villages where NECS was combined with IMAGINE infrastructure investments experienced a 10.3 percentage point positive impact on primary school enrollment, a 13.6 percentage point positive impact on attendance, a 0.21 standard deviation positive impact on local-language test scores, and no impact on French-language test scores. Due to the high cost of school construction and other investments undertaken by the projects, the economic rate of return (ERR) for the NECS-only project and the combined NECS and IMAGINE projects are estimated at 2% and -4%, respectively.

 

This evaluation is complete, and there are no planned next steps.

In Context

The MCC threshold program with Niger was planned as a three-year investment (March 2008 – September 2011) of $24.7 million (of program and administrative funding) in three projects: Girls’ Primary Education; Improve the Control of Corruption; and Streamline Business Creation and Land Access. The Improve the Education of Girls in Niger (IMAGINE) project and the subsequent Niger Education and Community Strengthening (NECS) project, which continued and complemented girls’ education activities begun under IMAGINE, are the subject of an independent impact evaluation released by MCC in July 2017, the results of which are summarized here.

 

The IMAGINE project, as originally designed, consisted of the construction of 68 primary schools with high quality infrastructure, along with implementation of a set of complementary interventions designed to increase girls’ enrollment and completion rates.

 

In December 2009, MCC suspended the Niger Threshold Program (NTP) due to a pattern of action by the Government of Niger (GoN) that was inconsistent with the democratic governance criteria MCC uses to select its partner countries. At the time of the suspension, $12 million of the planned $18.1 million budget for the IMAGINE project had been expended. The IMAGINE project was ultimately implemented in 10 departments in Niger, with 62 schools constructed. A few complementary activities, such as the provision of textbooks and materials for the schools, were fully implemented, whereas most planned complimentary activities were only partially implemented or not implemented at all.

 

In June 2011, MCC approved the reinstatement of threshold program assistance to Niger following the country’s return to democratic rule. In July 2012, the NTP was amended to reduce program funding to

$16.9 million, of which $2,000,000 was available to complete and expand the girls’ education component of the program. At the same time, USAID, using some of those funds as well as its own funds, agreed with the GoN to undertake the NECS project.

 

The objectives of NECS were twofold: (i) to increase access to quality education through a variety of investments including borehole construction and maintenance, community engagement and mentoring programs, and promotion of gender-equitable classrooms and student leadership activities; and (ii) to increase student reading achievement by implementing an ambitious early grade reading curriculum.

 

Program Logic

The interventions are listed in the left-hand column, with columns to the right listing the groups targeted by each activity and the outcomes that may plausibly improve in response to each activity.

The activities target a variety of groups in the community, including children, teachers, parents and  other adults, and school management committees. Together, the NECS and IMAGINE interventions were intended to foster improved school enrollment, attendance, and learning in the short term and perhaps improve longer-term outcomes such as employment and income once the children exposed to the interventions enter the workforce.

 

Groups directly

Activity                                        affected

Outcomes
Short term                           Medium term               Long term
Construct new girl- friendly schools** Students, especially girls Enrollment, attendance, learning  

 

 

 

 

 

 

 

 

Academic performance 1

 

 

 

 

 

 

 

 

 

Employment and income

Provide textbooks** Students Access to textbooks, learning
Introduce early-grade rapid reading program in local languages and provide teacher training and supervision Teachers, students Teaching techniques in early-grade reading in local languages, reading ability, learning; teacher capacity and accountability
Provide reading materials in local languages Students, adults in community Access to local-language reading materials, reading ability, learning
Develop mentoring program Students Enrollment, attendance, dropout rate, completion, learning
Promote gender- equitable classrooms Teachers, school management committees Girls’ enrollment, attendance, and learning
Promote leadership training for student government Students Student-teacher relations, student autonomy, self- esteem Attendance, student engagement, academic performance  

 

 

Employment and income

Support school management committees School management committees Community participation in education Quality of education, support for education
Establish adult literacy program Parents and adults in community Adult literacy, culture of reading Children’s enrollment, attendance, academic performance
Construct new boreholes* Students Access to safe drinking water Illness, attendance, retention General health, employment, and income
Facilitate general hygiene and sanitation Hand washing
Support deworming Deworming treatments
Key assumptions

·  Schools are sufficiently functional (for example, in terms of infrastructure and management) to support program interventions.

·  An adequate supply of teachers is available with the training and motivation to implement the early-grade reading curriculum.

·  The support provided by Ministry of Primary Education (MEP) inspectors and pedagogic supervisors is adequate to monitor implementation of the early-grade reading curriculum.

·  Sufficient participation and interest in other project activities will develop among key target groups (for example, adults in the community and school management committees).

·  No major disruptive events occur in the target villages (for example, famine or political unrest).

Note: No asterisk indicates an activity that was implemented solely as part of NECS. * indicates an activity that was originally implemented as part of IMAGINE and is being completed as part of NECS. ** indicates an activity that was implemented solely as part of IMAGINE.

 

Evaluation Questions

The evaluation was designed to answer the following questions:

  1. Sustainability of IMAGINE
    • Have the investments made under the IMAGINE project been sustained?

 

  1. Impact on key outcomes
    • What is the combined impact of the NECS and IMAGINE projects on key educational outcomes?
      • Primary education enrollment
      • Attendance rates
      • Learning as measured by test scores
    • What is the impact of just the NECS project on these key educational outcomes?

 

  1. Impact on subgroups
    • Do the estimated impacts differ for girls and boys?
    • Do the estimated impacts differ for children from households with different asset levels?

 

  1. Cost analyses
    • Was the NECS project investment justified from a cost perspective?
      • What was the project’s cost-effectiveness?
      • What was the project’s cost benefit?
      • What was the project’s economic rate of return (ERR)?

 

Evaluation Results

At the time of data collection in May – June 2016, most NECS activities had been fully implemented, with the exception of the construction of boreholes (52 of 59 were functional) and the introduction of the full early grade reading curriculum. The full curriculum was supposed to be implemented in both grades 1 and 2 during the 2015-2016 school year, but a high number of school disruptions (including teacher strikes, teacher absenteeism, late openings, and early closures) impeded completion of the academic year and implementation of the new curriculum in many schools. As of June 2016, only 15 percent of grade 1 classrooms and 33 percent of grade 2 classrooms were able to complete the full curriculum over the course of the 2015-2016 school year (Plan International, 2016). As a result, although children in  NECS intervention villages were exposed to most of the NECS activities for three years by the time of the data collection, their exposure to the early grade reading curriculum was more limited, so the estimated impacts of the package of intervention activities only reflects a partial implementation of the reading curriculum.

 

The original evaluation design provided for the collection of longitudinal data. However, delays in  project implementation as detailed above meant that many of the children in the 2013 sample may not have been exposed to the NECS reading curriculum at the time of data collection at the end of the 2015- 2016 school year. As a result, the decision was made to collect a repeated cross-section of households with children aged 6-12.

 

Evaluator Mathematica Policy Research
Impact or Performance? Impact
Methodology Random assignment

 

The design involved two rounds of clustered random assignment of villages within communes. The first round, conducted in 2008 for the IMAGINE evaluation, called for the random selection of IMAGINE treatment villages from a pool of potential recipient villages identified by the GoN according to a set of criteria (the remaining villages became the IMAGINE control villages). All IMAGINE recipient villages were selected to receive the NECS intervention. The second round of random assignment, conducted in 2012, involved the random selection of some of the IMAGINE control villages to receive NECS.

 

The final sample for this evaluation included 60 villages that received both IMAGINE and NECS, 82 villages that received only the NECS project, and 50 control villages.

 

The main sources of data were a household survey of randomly selected families with school-age children; the results of local- language and French-language reading tests and of math tests administered to children living in households interviewed in the household survey; a school survey administered to officials at the primary school in the village and direct observation of school infrastructure; and a village census used to select households with school-age children.

Evaluation Period Random assignment – IMAGINE*: December 2008 Implementation of IMAGINE: March 2009 – April 2010 First round of data collection: January – February 2011 Random assignment – NECS: November 2012

Second round of date collection**: October – November 2013 Implementation of NECS: June 2013 – May 2016

Third round of data collection: May – June 2016

 

* Random assignment occurred in December 2008, but the list of treatment villages was not finalized until February 2009 after completion of a ground-truthing exercise.

 

** The second round of data collection was originally scheduled to occur at the end of the 2012-2013 school year but was delayed until October – November 2013. Among the NECS project activities starting before the second round of data collection were training sessions related to gender and student recruitment efforts and the first of two adult literacy campaigns. However, activities directly related to learning outcomes did not start until after completion of the second round of data collection.

 

 Outcomes   Implementation of NECS Activities

  • By the end of the 2015-2016 school year, almost all schools in NECS & IMAGINE and NECS-only villages had an elected student government that had developed an action Those schools had, on average, 8.7 to 9.6 children in student government, with a nearly even split between boys and girls.
  • Schools in both treatment groups were about 60 to 65 percentage points more likely than schools in the control group to have established a mentoring program. Enrolled children in NECS-only villages are 7 percentage points more likely to report having a mentor, with children in NECS & IMAGINE villages not showing any impact on child-reported mentoring.
  • The majority of treatment schools reported that they had on hand local-language story books, and 47 to 60 percent of schools reported that they used the books for classroom Almost no control group schools reported that they had instituted a local-language reading curriculum or used local-language materials.
  • In the NECS & IMAGINE and NECS-only villages, 3 percent and 11.6 percent of households, respectively, noted that an adult member had participated in literacy training in the past year compared to just 2.9 percent of households in control villages. Adults in treatment communities were also about 3 to 5 percentage points more likely than adults in control villages to have participated in community events related to literacy and reading in the past year.

Sustainability of IMAGINE Infrastructure Investments

  • The number of overall classrooms in IMAGINE schools and those with finished materials had increased from 6.5 to 2 and from 5.0 to 6.9, respectively, between 2013 and 2016.
  • The presence in IMAGINE schools of separate latrines for girls and boys and housing for female teachers declined by

24.3 percentage points and 19.4 percentage points, respectively, between 2013 and 2016. However, the presence of latrines and housing in IMAGINE schools did not change during that timeframe.

  • The likelihood of having a functioning potable water source at IMAGINE schools improved from 50 percent in 2013 to

75.4 percent in 2016, which is consistent with the planned borehole construction and rehabilitation activities implemented under the NECS project.

School Enrollment

  • Children in NECS & IMAGINE villages and NECS-only villages were 3 percentage points and 9.5 percentage points more likely to report school enrollment in the current school year than children in control group villages, respectively.

Absenteeism

·         Children in NECS & IMAGINE villages and NECS-only villages were 13.6 percentage points and 11.1 percentage points more likely to report school attendance on the most recent day the school was open than children in control group villages, respectively.

Objective-level Outcomes Learning Outcomes – Local Language

·         The average normalized local-language test score was 0.21 standard deviations and 0.15 standard deviations higher for children in NECS & IMAGINE villages and NECS-only villages, respectively.

Learning Outcomes – French

·         Test scores in French were higher for children in treatment villages but the difference was not statistically significant.

Learning Outcomes – Math

·         Positive impacts of 0.13 and 0.10 standard deviations were detected on standardized math test scores in NECS & IMAGINE and NECS-only villages, respectively.

Differential Impacts on Sub-groups

·         The difference in impacts between girls and boys is not statistically significant for enrollment, attendance, and local- language test scores, except for local-language test scores in the NECS & IMAGINE group, where the impacts were higher for boys.

·         The evaluation found no significant differences in impacts between the socioeconomic groups, except for local- language test scores in the NECS-only group, where impacts were higher for the lowest quintile.

Cost Analyses

·         The combination of the NECS & IMAGINE projects is less cost-effective than the NECS-only project for enrollment,

attendance, and local-language test scores because the NECS and IMAGINE projects achieved highly similar impacts; however, the IMAGINE project involved school construction, which is significantly more expensive.

·         The economic rate of return (ERR) for the NECS & IMAGINE projects and NECS-only project are estimated at -4% and 2%, respectively.

Effect on household income attributable to MCC The evaluation was not designed to assess the impact of the project on employment and incomes; moreover, it is likely too early to do so.

 

Lessons Learned

The main lessons from the project are:

  • Impacts of IMAGINE and NECS projects do not appear to be additive: The results suggest that both the IMAGINE project and NECS project had positive impacts on most educational

 

outcomes. The impacts of the two projects are largely similar to one another and similar to the impacts of the IMAGINE project observed in the three-year evaluation. However, it is uncertain whether the combination of the two projects had additional benefits on enrollment,  attendance, or mother tongue early grade reading skills on children of primary school age beyond the benefits of each program alone. It is possible that the two projects did not have additive benefits when implemented together. For example, the extensive social mobilization campaign that occurred as part of the NECS project may have improved child outcomes in NECS- only villages but may not have had an additional benefit in villages that had already experienced similar benefits from the IMAGINE project. It is also possible that the benefits of IMAGINE declined over time (for example, if enrollment and attendance in non-IMAGINE villages “caught up” over time), and the NECS project had similar impacts on both sets of communities.

 

  • School infrastructure improvements and complimentary investments have broader learning impacts: The evaluation observed a positive impact on math test scores for both NECS & IMAGINE and NECS-only villages. This is an important educational outcome that was not directly targeted by the projects and may have resulted from improvements in school attendance, teaching quality, or other aspects of the learning environment.

 

  • The cost of education investments should account for the labor market and economic realities that project beneficiaries will face: Despite achieving statistically significant positive impacts on enrollment, attendance, and learning outcomes, the projects produced low, and in the case of NECS & IMAGINE, economic rates of This was, in large part, due to the high cost of the investments and the low economic returns to education in Niger’s labor market. While the low returns to education contribute to the low economic rates of return of the project, they also provide important insight into the demand for education, as children and their families may see little advantage to school enrollment or pursuing an education. The projects were able to improve the perceptions of education in the treatment villages – parents in NECS & IMAGINE and NECS-only villages were 6.9 percentage points and 6.5 percentage points, respectively,  more likely to want their child to attend secondary or higher schooling. However, further improvements in parental perceptions about education and further household investments in education may both be hampered by parents’ observations of low labor market returns to education. Working to find lower-cost solutions that provide access to quality education may be a promising area for future programming for the GoN and other donors working in Niger. Moreover, the demand for education and the economic and employment prospects of those targeted by the investment should be explicitly accounted for when designing education interventions in environments such as Niger.

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. Academic performance refers to completion of primary school and subsequent levels of schooling, and to test scores and grades in coursework.