Program OverviewMCC’s $140 million Georgia II Compact (2014-2019) funded the $73 million Improving General Education Quality (IGEQ) Project, which aimed to improve the quality of public Science, Technology, Engineering, and Mathematics (STEM) education in grades 7-12. The IGEQ Project invested in rehabilitating education infrastructure and constructing science laboratories in targeted schools. A one-year sequence of training activities was provided to STEM educators and school directors on a nationwide basis. Download Georgian translated evaluation brief.
MCC commissioned Mathematica to conduct an independent interim performance evaluation of the IGEQ Project. Full report results and learning: https://data.mcc.gov/evaluations/index.php/catalog/151.
- In the first phase of school rehabilitation (29 schools), students experienced large improvements compared to baseline in heating, lighting, sanitation, building quality, and access to science laboratories and recreation facilities.
- Students and teachers agreed that these improvements addressed barriers to using classroom time effectively on instruction.
- The final report will estimate impacts for all rehabilitated schools and measure whether infrastructure upgrades improved learning outcomes.
- The teacher training component was successfully delivered on a nationwide scale, with high completion rates for school directors (93 percent) and teachers (82 percent).
- One month after the one-year training sequence concluded, teachers reported that they had improved confidence using student-centered teaching practices, and school directors reported that they had increased provision of instructional leadership. However, there was little evidence of immediate changes in teachers’ classroom instruction practices.
- Planners of the teacher training component anticipate that further changes in instructional practices could develop over time. The final report will examine trends in teaching practices several years after the training sequence ended.
Evaluation QuestionsThis interim performance evaluation was designed to assess the impact of infrastructure and training improvements on STEM education by answering the following questions:
- 1 Did school rehabilitation deliver improved facilities?
- 2 What are the impacts of rehabilitation on the school environment, including temperature, lighting, equipment, and infrastructure maintenance?
- 3 What are the perceptions of students, parents, teachers, and school directors about the effects of rehabilitation on safety, comfort, and the extent to which time in school is used effectively for learning?
- 4 Did training initiatives for teachers and school directors succeed in delivering training on a nationwide basis?
- 5 To what extent do school directors perceive that their instructional leadership and school management skills have changed as a result of the new training intervention?
- 6 To what extent do teachers perceive that their pedagogical and classroom management practices have changed as a result of the new training intervention?
- 7 Did teacher training modules improve teachers’ knowledge of and willingness to use practices related to student-centered instruction, formative assessments, and improved classroom management?
School RehabilitationSchool Rehabilitation Delivered Large Improvements in Infrastructure The first phase of school rehabilitation delivered infrastructure improvements to 29 schools (later phases plan to reach a total of up to 96 schools). The improvements were readily visible. At baseline, most of the schools had at least one classroom with two or more problematic conditions present (such as cracks, water damage, mold, chipped or peeling paint, or holes in ceilings and floors). But after rehabilitation, the percentage of schools with two or more problems in at least one classroom dropped from 72 to 7 percent for ceilings and from 72 to 0 percent for floors. At baseline, about half of all observed classrooms did not have functional central heating and 80 percent of schools lacked functional electric lighting. After rehabilitation, all of these schools had an operational central heating system and the percentage of classrooms with functional lighting systems improved by nearly 60 percentage points. Rehabilitated schools consistently received flush toilet facilities (lacking in 83 percent of schools at baseline), new science laboratory facilities (lacking in half of schools at baseline), and improved indoor recreation facilities as well. Improvements Addressed Barriers to Using Classroom Time Effectively for Instruction Students and teachers agreed that these infrastructure improvements addressed barriers to instruction. For example, after rehabilitation the percentage of students who felt uncomfortable due to inadequate winter heating fell from 41 to 6 percent, the percentage of teachers who felt that classroom lighting was inadequate fell from 29 to 4 percent, and student comfort with sanitation facilities improved dramatically. Teachers also reported high levels of satisfaction with access to new science laboratory facilities for instruction. However, school directors did report struggling with substantial increases in utility costs for heating and lighting, and it remains to be seen whether improved facilities will be maintained as planned. This evaluation’s final report will assess whether rehabilitation ultimately produced improvements in student learning across all rehabilitated schools, and will measure the effects of rehabilitation on school operating costs and maintenance practices over two years.
Educator TrainingThe Program Delivered Training to Teachers and School Directors on a Nationwide Scale In total, the program succeeded in offering training to Georgia’s entire population of secondary-level school directors (about 2,000) and all of Georgia’s upper-grade teachers in the subjects of science, mathematics, English, and geography (about 18,000 teachers in total). The training sequence consisted of multiple modules, and was held for each cohort over the course of about one year. Attendance rates at the trainings were generally high. Although school directors completed the full training sequence at a higher rate (93 percent) than teachers in the first cohort (82 percent) or second cohort (55 percent at the time of this report, when makeup trainings were still being held), a large majority of both groups attended at least one training session, and nearly all of the trainees felt positively about the training experience. Teachers and Directors Reported Improved Instructional Confidence After Training After the training sequence concluded, school directors reported that the training improved their capacity to provide instructional leadership through curriculum guidance, classroom observation, and supporting teachers’ professional development. Teachers also became more confident in their ability to teach higher-order thinking skills, promote cooperation through group work, and use lesson plans that include formative assessments and differentiated instruction for students with different abilities. However, there was little evidence of immediate changes in teachers’ classroom instruction practices, and in focus groups some teachers voiced concerns about the amount of time and effort needed to implement these practices consistently. Planners of the teacher training component anticipate that further changes in instructional practices could develop over time. To assess whether these changes occur in practice, this evaluation’s final report will examine trends in teaching practices several years after the training sequence ended.
- Construction timelines for the Improved Learning Environment Infrastructure (ILEI) Activity were significantly longer than originally anticipated. For future school infrastructure investments, MCC should stress realistic work planning from the start of the Compact.
- Stakeholder input is critical to survey module design and made the Training Teachers for Excellence (TEE) Activity data more useful to both MCC and the Ministry of Education, Science, Culture and Sport. Future evaluations should plan time for a survey design workshop in country.
- The success of the TEE Activity in reaching its participation targets across Georgia was due to a great deal of collaboration between the MCA-Georgia, Program Management Unit within the Teacher Professional Development Centre, and Program Management Consultant, IREX. Conduct adequate due diligence to understand the cost and implementation requirements on MCA-Georgia and implementing partners.
- The ILEI evaluation’s use of pairwise randomization implemented by construction phase helped mitigate risks to the evaluation due to uncertainties around the number of treatment schools and construction timelines. MCC should build flexibility into evaluation design whenever possible.