MCC’s $140 million Georgia II Compact (2014-2019) funded the $73 million Improving General Education Quality (IGEQ) Project, which aimed to improve the quality of public Science, Technology, Engineering, and Mathematics (STEM) education in grades 7-12. The IGEQ Project invested in rehabilitating education infrastructure and constructing science laboratories in targeted schools. A one-year sequence of training activities was provided to STEM educators and school directors on a nationwide basis.
MCC commissioned Mathematica to conduct an independent interim performance evaluation of the IGEQ Project. Full report results and learning: https://data.mcc.gov/evaluations/index.php/catalog/151.
- In the first phase of school rehabilitation (29 schools), students experienced large improvements compared to baseline in heating, lighting, sanitation, building quality, and access to science laboratories and recreation facilities.
- Students and teachers agreed that these improvements addressed barriers to using classroom time effectively on instruction.
- The final report will estimate impacts for all rehabilitated schools and measure whether infrastructure upgrades improved learning outcomes.
- The teacher training component was successfully delivered on a nationwide scale, with high completion rates for school directors (93 percent) and teachers (82 percent).
- One month after the one-year training sequence concluded, teachers reported that they had improved confidence using student-centered teaching practices, and school directors reported that they had increased provision of instructional leadership. However, there was little evidence of immediate changes in teachers’ classroom instruction practices.
- Planners of the teacher training component anticipate that further changes in instructional practices could develop over time. The final report will examine trends in teaching practices several years after the training sequence ended.
This interim performance evaluation was designed to assess the impact of infrastructure and training improvements on STEM education by answering the following questions:
Did school rehabilitation deliver improved facilities?
What are the impacts of rehabilitation on the school environment, including temperature, lighting, equipment, and infrastructure maintenance?
What are the perceptions of students, parents, teachers, and school directors about the effects of rehabilitation on safety, comfort, and the extent to which time in school is used effectively for learning?
Did training initiatives for teachers and school directors succeed in delivering training on a nationwide basis?
To what extent do school directors perceive that their instructional leadership and school management skills have changed as a result of the new training intervention?
To what extent do teachers perceive that their pedagogical and classroom management practices have changed as a result of the new training intervention?
Did teacher training modules improve teachers’ knowledge of and willingness to use practices related to student-centered instruction, formative assessments, and improved classroom management?
School Rehabilitation Delivered Large Improvements in Infrastructure
The first phase of school rehabilitation delivered infrastructure improvements to 29 schools (later phases plan to reach a total of up to 96 schools). The improvements were readily visible. At baseline, most of the schools had at least one classroom with two or more problematic conditions present (such as cracks, water damage, mold, chipped or peeling paint, or holes in ceilings and floors). But after rehabilitation, the percentage of schools with two or more problems in at least one classroom dropped from 72 to 7 percent for ceilings and from 72 to 0 percent for floors. At baseline, about half of all observed classrooms did not have functional central heating and 80 percent of schools lacked functional electric lighting. After rehabilitation, all of these schools had an operational central heating system and the percentage of classrooms with functional lighting systems improved by nearly 60 percentage points. Rehabilitated schools consistently received flush toilet facilities (lacking in 83 percent of schools at baseline), new science laboratory facilities (lacking in half of schools at baseline), and improved indoor recreation facilities as well.
Improvements Addressed Barriers to Using Classroom Time Effectively for Instruction
Students and teachers agreed that these infrastructure improvements addressed barriers to instruction. For example, after rehabilitation the percentage of students who felt uncomfortable due to inadequate winter heating fell from 41 to 6 percent, the percentage of teachers who felt that classroom lighting was inadequate fell from 29 to 4 percent, and student comfort with sanitation facilities improved dramatically. Teachers also reported high levels of satisfaction with access to new science laboratory facilities for instruction. However, school directors did report struggling with substantial increases in utility costs for heating and lighting, and it remains to be seen whether improved facilities will be maintained as planned. This evaluation’s final report will assess whether rehabilitation ultimately produced improvements in student learning across all rehabilitated schools, and will measure the effects of rehabilitation on school operating costs and maintenance practices over two years.
The Program Delivered Training to Teachers and School Directors on a Nationwide Scale
In total, the program succeeded in offering training to Georgia’s entire population of secondary-level school directors (about 2,000) and all of Georgia’s upper-grade teachers in the subjects of science, mathematics, English, and geography (about 18,000 teachers in total). The training sequence consisted of multiple modules, and was held for each cohort over the course of about one year. Attendance rates at the trainings were generally high. Although school directors completed the full training sequence at a higher rate (93 percent) than teachers in the first cohort (82 percent) or second cohort (55 percent at the time of this report, when makeup trainings were still being held), a large majority of both groups attended at least one training session, and nearly all of the trainees felt positively about the training experience.
Teachers and Directors Reported Improved Instructional Confidence After Training
After the training sequence concluded, school directors reported that the training improved their capacity to provide instructional leadership through curriculum guidance, classroom observation, and supporting teachers’ professional development. Teachers also became more confident in their ability to teach higher-order thinking skills, promote cooperation through group work, and use lesson plans that include formative assessments and differentiated instruction for students with different abilities. However, there was little evidence of immediate changes in teachers’ classroom instruction practices, and in focus groups some teachers voiced concerns about the amount of time and effort needed to implement these practices consistently. Planners of the teacher training component anticipate that further changes in instructional practices could develop over time. To assess whether these changes occur in practice, this evaluation’s final report will examine trends in teaching practices several years after the training sequence ended.
Construction timelines for the Improved Learning Environment Infrastructure (ILEI) Activity were significantly longer than originally anticipated. For future school infrastructure investments, MCC should stress realistic work planning from the start of the Compact.
Stakeholder input is critical to survey module design and made the Training Teachers for Excellence (TEE) Activity data more useful to both MCC and the Ministry of Education, Science, Culture and Sport. Future evaluations should plan time for a survey design workshop in country.
The success of the TEE Activity in reaching its participation targets across Georgia was due to a great deal of collaboration between the MCA-Georgia, Program Management Unit within the Teacher Professional Development Centre, and Program Management Consultant, IREX. Conduct adequate due diligence to understand the cost and implementation requirements on MCA-Georgia and implementing partners.
The ILEI evaluation’s use of pairwise randomization implemented by construction phase helped mitigate risks to the evaluation due to uncertainties around the number of treatment schools and construction timelines. MCC should build flexibility into evaluation design whenever possible.
This mixed-methods interim evaluation was conducted toward the end of the Compact, when program implementation was almost complete. It drew on a combination of infrastructure assessments, surveys, and qualitative data collected from students, parents, teachers, and school directors.
School Rehabilitation Study: Before rehabilitation activities began, a local data collection firm conducted a baseline data collection round assessing school infrastructure conditions (including direct measurements of heating, lighting, air quality, and building safety) and collecting survey data from students, parents, teachers, and school directors. The interim study focused on the first 29 rehabilitated schools, and collected follow-up data in these schools during the winter of 2017 and 2018, in the first year after rehabilitation was completed. The study also conducted student focus groups and in-depth interviews with a sub-sample of teachers and school directors to learn more about how rehabilitation changed the learning environment. As part of the final evaluation report, the study will estimate the impacts of rehabilitation using a randomized control trial design, comparing learning outcomes in rehabilitated schools to outcomes in a control group of schools that were not rehabilitated.
Teacher and School Director Training Study: The interim analyses relied primarily on post-training surveys conducted with 1,186 teachers and 119 school directors in the first month after the one-year training sequence was completed (the surveys took place in 2017 and 2018). While this outcomes study was primarily descriptive in nature, for a subset of teachers the study included a matched comparison group analysis to estimate the initial effects of training on teachers’ knowledge and classroom practices. In addition, the evaluation included qualitative focus groups with teachers, in-depth interviews with trained school directors, and a validation exercise comparing teachers’ self-reported survey data to survey data from students and classroom observation data for a small sub-sample of teachers to assess if teachers’ survey responses are consistent with other data sources. The final report will include longer-term analyses examining trends in the post-training outcomes of teachers and school directors over several years.
A final study covering the full set of evaluation questions is underway. Results will be available in 2021.