Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

  • Final Evaluation Brief
  • February 2024

Strengthening Secondary Education in Guatemala

Investments in teachers, parents, and information systems improved quality

View as PDF

Program Overview

MCC’s $27.3 million Guatemala Threshold Program (2016-2021) aimed to provide Guatemala’s youth with quality educational opportunities that have relevance to the labor market and mobilize additional government resources that are needed to address binding constraints to economic growth. The Education Project consisted of three activities: (1) Quality of Education in Support of Student Success; (2) Improving Technical and Vocational Education and Training; and (3) Strengthening of Institutional and Planning Capacity. This Evaluation Brief focuses on Activities 1 and 3.

Evaluator Description

MCC commissioned Mathematica to conduct independent impact and performance evaluations of Activity 1 and a performance evaluation of Activity 3 to estimate impacts on student learning, teacher recruitment and selection, and the use of data for decision-making. Full final report: https://evidence.mcc.gov/evaluations/index.php/catalog/216.

Key Findings

Teacher Training, Recruitment and Selection

  • Activity 1, known as Éxito Escolar, was a comprehensive set of interventions to strengthen Ciclo Básico, or lower-secondary education, in five regions of Guatemala.
  • Teacher competency improved in the treatment schools but also did in the control schools.
  • Teachers’ perceptions of the professional development, improved student learning and Ciclo Básico programs were overwhelmingly positive. However, there was no impact on student test scores.
  • Stakeholders felt that the parent organizations supported schools and are implementing aspects of the action plans.
  • The Guatemalan Ministry of Education approved a new teacher recruitment and selection system but did not implement it due to COVID and a lack of resources.

Service Standards for Planning and Budgeting

  • Although spending on lower secondary education increased during the Education Project, most education stakeholders do not perceive that spending has increased.

Decision Support Systems for Planning and Budgeting

  • Study participants agree that the distribution of resources does not respond to the human and material requirements for implementing the national curriculum. What data exists is rarely used to support decision-making.

Evaluation Questions

This final impact and performance evaluation was designed to answer the following questions:

  1. 1
    To what extent were Éxito Escolar’s planned activities for teacher professional development, pedagogical support, and development of parent organizations and school networks implemented as designed? What impact did the interventions have on student learning?
  2. 2
    What were the main facilitators of and barriers to implementing Éxito Escolar activities in terms of reaching hypothesized medium-term outcomes including:
    a. Improvements in student learning
    b. Higher graduation rates
    c. Increased retention and promotion of secondary education students
  3. 3
    Were the Institutional Strengthening activities implemented as planned (if not, why)? To what extent did the project lead to changes in budget for secondary education? To what extent did the project lead to changes in teacher hiring/retention?

Detailed Findings

Teacher Training, Recruitment and Selection

PADEP/Ciclo Basico enrollment targets and actual enrollment. 1,913 teachers and directors were enrolled, and1,545 completed, PADEP/CB versus an original target of 2,400 and an updated target of 1,800.

PADEP/Ciclo Basico enrollment targets and actual enrollment

Éxito Escolar was a comprehensive set of interventions to strengthen Ciclo Básico education in five regions of Guatemala. The activity included training of teachers and school directors, and engaging parents to strengthen school leadership and raise awareness of the importance of Ciclo Básico. In interviews and focus groups, teachers, school directors, and pedagogical advisors shared that they perceived that teachers were using the innovative methods and that those methods were motivating students to remain in school. Teachers expected lower dropout rates, higher promotion rates, and lower grade repetition. Evidence pointed to improved teacher competency in the treatment schools. However, this was also the case for teachers in the control schools. Stakeholders felt that the parent organizations were supporting schools to implement aspects of the action plans. Management advisors and school directors reported that they were able to implement at least a portion of the action plans. However, the interventions did not have an impact on student learning outcomes. It is likely that the COVID pandemic caused further learning loss in students who were already behind grade level making it difficult for Éxito Escolar to overcome learning loss in treatment schools. Differences in estimated impacts were not significantly different between male and female students for language or science, but were significantly lower for female students than male students in math (p = 0.018).

The Guatemalan government officially adopted a Ministerial Agreement to implement the new teacher recruitment and professional development process developed in January 2021. However, hiring was suspended during COVID and has not restarted as of August 2023, which stalled implementation of the new system. The Ministry is using one of the project’s data systems which registers and calculates the demand for teachers, but other capacity building systems are not in use, including those to support teacher recruitment, selection, and monitoring.

Percentage of teachers who engage in each behavior more than 75 percent of class time. Manages classtime effectively: 42%. Effectively use different instructional resources and strategies to explain concepts: 36%. Engages students in carefully constructed active learning experiences: 35%. Provides students with consistent and comprehensive feedback: 30%. Provides opportunities for students to consider how learning concepts apply to their everyday life: 26%. Provides students with opportunities to practice critical thinking and higher-order thinking: 19%.

Classroom observations confirmed that teachers in the treatment group used the new teaching methods in their classrooms.

Service Standards for Planning and Budgeting

The project identified seven learning standards, but had mixed results. For example, stakeholders do not expect meaningful changes in budgeting. While administrative data shows that the budget for Ciclo Básico increased relative to per-pupil primary school spending, the increase is unlikely related to the institutional strengthening activity. Funding for two high-priority learning standards increased, but it is unclear to what extent the activity contributed to these changes. Despite these small gains, low investment in Ciclo Básico continues to be a structural barrier in Guatemala.

Decision Support Systems for Planning and Budgeting

The project created 14 new information systems to support the use of data to drive decision-making, but the Ministry is currently only using one. Data is rarely used to support decision-making. Most study respondents agree that the distribution of education resources does not respond to the human and material requirements for implementing the national curriculum. Respondents also indicated that information to recruit and select teachers, ensure that students have access to adequate school facilities, and manage learning materials is accessed “with a lot of effort.”

Barriers to institutionalizing reforms include: lack of staff capacity to run, maintain and use the information systems; lack of prioritization of Ciclo Básico; and the absence of a “data champion” within the Ministry to institutionalize the use of data and evidence in policy.

MCC Learning


  • Offering varied modalities of programming, including virtual or hybrid instruction, can improve take-up and participation rates in teacher training programs.

  • Teacher training programs should consider the competing demands on teachers' time and provide adequate incentives and support relevant to their work.

  • Consider early on how to incentivize and build momentum for partner country institutional reforms.

  • For student assessments to accurately discern changes in student performance, the difficulty of the questions needs to be aligned with average student performance in the context they are used.

Evaluation Methods

For activity 1, the implementation study had three rounds of key informant interviews and focus group discussions. The RCT measured impacts on teachers' time on task, student learning, and enrollment outcomes. It also measured student learning and enrollment outcomes.
<p>For activity 3, the fidelity of implementation evaluation used multiple rounds of key informant interviews with various stakeholders. It examined factors affecting the design, implementation, and results with system data and document review. The PEA had multiple rounds of key informant interviews with various stakeholders. It also had Delphi surveys to systematically gather and consolidate opinions." width="100%" height="" /></figure></p>
<p>Activity 1 is a is a mixed-methods evaluation organized into two main components. The first component is an implementation study based on document review and three rounds of key informant interviews and focus group discussions with stakeholders. The first round was conducted in Spring 2019, the second round was conducted in 2020 and 2021, and the third round of qualitative data collection was done in 2022. The second component is a randomized controlled trial (RCT) to estimate Éxito Escolar’s impacts on teachers’ time on task, student learning, and enrollment outcomes. Student learning was measured with a student assessment and enrollment outcomes are captured through the Ministry of Education’s administrative data.</p>
<p>The Activity 3 evaluation uses a mixed-methods approach to conduct a performance evaluation composed of two main components: fidelity of implementation and political economy analysis. The Fidelity of Implementation Study draws on multiple rounds of key informant interviews with national and departmental staff, staff from the implementing organization, and staff of the Accountable Entity, PRONACOM. The study also draws on education management information system data, and a document review to examine factors affecting the design, implementation, and results of Activity 3.</p>
<p>The Political Economy Analysis (PEA) uses a mixed-methods approach. The qualitative approach to this analysis draws on multiple rounds of key informant interviews with national and departmental staff and staff of FHI360, the implementing entity, to gather information that allows us to employ DFID’s Drivers of Change framework during analysis (Warrener 2004). The quantitative approach draws on Delphi surveys to systematically gather and consolidate opinions about the system, its features and status, prior to the COVID-19 pandemic. </p>

                                
                                
                                    <p class=2024-002-2942