Summary of Findings

Measuring Results of the El Salvador Formal Technical Education Sub-activity

View as PDF

  • The first MCC compact with El Salvador was a five-year investment (2007-2012) of $449.6 million. The Formal Technical Education Sub-Activity is the subject of three independent impact and performance evaluations summarized here.
     The combined components of the sub-activity were expected to increase the number of skilled workers and provide labor insertion services, which were ultimately expected to increase employment of participants and lead to an increase in overall household income.
  • Although the Strengthening of 20 Secondary Schools and Scholarships to Technical Secondary Schools increased enrollment, especially among boys, they showed no impact on income one year after students were scheduled to complete general or technical degrees, due in part to increased enrollment in post-secondary education. As for the Strengthening of the Chalatenango Technical Institute (ITCHA), employment rates one year after graduation fell below the compact target of 70%. This was partially due to students’ further engagement in university studies, but also likely due to low labor market demand in the region.
  • The below target employment rates and lack of impacts on income demonstrate a valuable lesson that involvement of the private sector is an important aspect of providing quality education and training. No matter how successful the implementation of a project is, there may not be enough employment opportunities for newly qualified secondary school and ITCHA graduates.
  • This evaluation is complete and there are no planned next steps.

In Context

The first MCC compact with El Salvador was a five-year investment (2007-2012) of $449.6 million in 3 projects: connectivity, human development and productive development. Implemented by a unit set up by the Government of El Salvador (called FOMILENIO), the Human Development Project included two major activities, Education and Training and Community Development. The $25.8 million Education and Training Activity consisted of three Sub-Activities: Technical Assistance, Formal Technical Education, and Non-Formal Skill Development. The Formal Technical Education Sub-Activity is the subject of three independent impact and performance evaluations released by MCC, the results of which are summarized here. The evaluations cover a portion of the Education and Training Activity and the entire Activity represents 5.7 percent of the total compact. Other components of the compact are the subject of both completed and forthcoming independent evaluations.

Graph: Pie chart of expenditures in the El Salvador Compact

Program Logic

The program logic for the Formal Technical Education Sub-Activity assumed that a package of interventions, including improved infrastructure and equipment, scholarships, teacher training, and new curricula would reduce barriers for students to enroll in 20 selected technical secondary schools and improve the quality of education. It was expected that the improved secondary schools would supply the post-secondary Chalatenango Technical Institute (ITCHA) with students who were better prepared for post-secondary education. At the same time, ITCHA underwent significant improvements (in infrastructure and curricula) to transform it into a Gradual Educational Model of Technical and Technological Learning (known as MEGATEC for its initials in Spanish) institution. MEGATEC is an educational model based on the premise that technical education should be tailored to regional economies’ labor market demands and structured to impart key competencies required of professionals in technical fields. Within the MEGATEC model, new curricula would follow a competency-based methodology for which teachers would be specially hired and trained.

Although not included in the program originally, the Sustainable Labor Insertion Program (knows as PILAS for its initials in Spanish) was added during implementation as a complement to the compact’s educational investments. PILAS was designed to link trained individuals, including secondary school and ITCHA graduates, to employment opportunities, which included either support starting their own business or employment with an existing enterprise. Together, the expected increase in skilled workers plus labor insertion services were expected to increase employment of participants, which was expected to lead to an increase in overall household income.

Chart: Flow chart articulating the program logic for the project

There were several key assumptions underlying the Formal Technical Education Sub-Activity program logic during the design of the investment:

  • Lack of capacity of schools and lack of student funds were the primary reasons for lower enrollment
  • Current educational quality was low and new technical curricula would teach skills demanded by employers
  • Sufficient job opportunities exist for additional graduates of technical education
  • The Ministry of Education of El Salvador will continue to maintain infrastructure, continue to provide scholarships, and continue to train teachers after the compact to ensure sustainability

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators Level Baseline
(2007)
Actual Achieved
(09/2012)
Target Percent Complete
Formal Technical Education Overall
Graduates from MCC-supported education activities Outcome 0 4,285 6,795 63%
Students participating in MCC-supported education activities Outcome 0 30,672 26,927 114%
Instructors and administrators trained Output 0 566 500 113%
Formal Technical Education – Technical Secondary Schools
Graduation rate of technical secondary schools Outcome 71% 67% 71% 94%
Enrolled students of technical secondary schools Outcome 7,600 9,720 9,413 117%
Scholarships granted to students at the technical secondary schools Output 0 3,409 3,445 99%
Technical secondary schools remodeled and equipped Output 0 20 20 100%
Formal Technical Education – Chalatenango Technical Institute
Graduation rate of the Chalatenango Technical Institute Outcome 73% 89% 73% 122%
Enrolled students of the Chalatenango Technical Institute Outcome 264 613 540 126%
Scholarships granted to students for post-secondary technical education Output 0 921 932 99%
Chalatenango Technical Institute remodeled and equipped Output 0 1 1 100%

Source: Closeout ITT from December 2012, which includes data through the end of the compact
The average completion rate of output and outcome targets is 104 percent; and in 7 of the 11 indicators, targets were met or exceeded. 1

Evaluation Questions

The evaluations of the Formal Technical Education Sub-Activity were customized for each component within the Sub-Activity and were designed to answer different questions. However, because the secondary scholarship program was designed to work in conjunction with secondary school strengthening investments in 20 pre-selected secondary schools and therefore had a shared target population and objectives, the evaluation of those two components addressed a common set of research questions.

Strengthening of 20 Technical Secondary Schools and Scholarships for Technical Secondary Schools

  • How were the secondary school strengthening and scholarship programs designed and implemented? Did implementation meet original targets and expectations?
  • What are the characteristics (age, gender, initial household income, and so on) of scholarship recipients and secondary school students? What are students’ professional aspirations and constraints to education and employment?
  • What is the impact of FOMILENIO’s program for strengthening secondary school on students’ education and labor market outcomes, including secondary school enrollment, grade completion, graduation, additional education, employment, and income? What is the impact on student educational and labor outcomes of the offer of scholarships in some programs within strengthened schools?
  • Were impacts different for girls versus boys? Did some groups experience positive or negative outcomes relative to other groups?
  • What aspects of implementation can provide context for understanding impact findings? Can socioeconomic factors or elements of implementation help explain (potential) differences in impacts for girls versus boys?
  • Are secondary school improvements and scholarships being maintained? Are schools well positioned to provide students in the region with a high quality secondary technical education in future years?

Strengthening of the Chalatenango Technical Institute

  • How were ITCHA strengthening efforts designed and implemented? Did implementation meet original targets and expectations? How were MEGATEC degree programs implemented at ITCHA and its linked secondary schools?
  • What are the characteristics (age, gender, income, and so on) of ITCHA students? What are students’ professional aspirations and constraints to education and employment?
  • Did enrollment, graduation, and employment outcomes meet stakeholders’ expectations? Why or why not?
  • Who was likely to benefit most from the ITCHA/MEGATEC investments? Were results different for girls versus boys?
  • What are potential reasons that results did or did not meet expectations? If results were different for girls versus boys, why?
  • Are the improvements to ITCHA and scholarships being maintained? Is ITCHA well positioned to provide students in the region with a high quality postsecondary technical education in future years?

Evaluation Results

The Formal Technical Education Sub-Activity took the form of three separate evaluations, the results of which can be separated out into three components: (1) Strengthening of 20 Technical Secondary Schools, (2) Scholarships for Technical Secondary Schools 2 , and (3) Strengthening of the Chalatenango Technical Institute. The evaluations used multiple methodologies in order to answer different research questions. The evaluation results for each component and each methodology are discussed in turn.

Implementation Findings for the 20 Technical Secondary Schools

The objective of the implementation portion of the evaluation was to document how the components of the sub-activity were designed and assess if they were implemented as planned.

Strengthening of 20 Technical Secondary Schools
Evaluator Mathetmatica Policy Research
Impact or Performance? Performance
Methodology Mixed methods
Evaluation Period Interviews in 2011, 2014 and 2015; Administrative records reviewed from 2007 to 2015
Outcomes
  • Stakeholders selected 20 needy schools for new academic programs, capital improvements, and teacher training. Principals and students were satisfied with infrastructure improvements, with some exceptions. A minority of principals said that they would have liked to have been more involved in the design and execution of infrastructure investments.
  • Four secondary schools introduced MEGATEC degree programs in 2010. Ten strengthened schools introduced certificate programs. Stakeholders expressed appreciation for the new degree and certificate programs, although some had reservations about the alternative tourism curriculum.
  • CIDE designed and conducted teacher and stakeholder training focused on competency-based education. FOMILENIO representatives and school staff reported high satisfaction with the training offered by CIDE.
  • The Ministry of Education (MINED) provided some ad hoc follow-up post compact, but stakeholders would have liked more support.
Objective N/A
Effect on household income attributable to MCC N/A

 

Impact Findings for the Strengthening of 20 Technical Secondary Schools

The impact evaluation of the Strengthening of the 20 Technical Secondary Schools assessed whether the intervention improved educational and labor market outcomes for students attending schools affected by the intervention. The educational outcomes of interest include enrollment, grade completion, on-time grade progression, and academic achievement; while labor market outcomes include employment, income, and continuation in post-secondary education. It should be noted that the scholarship program for secondary education was designed to work in conjunction with investments to strengthen 20 pre-selected secondary schools. Due to the shared target population and objectives of these two interventions, and the fact that 17 of the 20 treatments schools and none of the 20 comparison schools received scholarships, the evaluation results may not be solely attributable to the MCC funded project , and instead is a reflection of both interventions working together. For instance, increase in enrollment is likely a result of both increasing students’ demand for technical education (primarily through scholarships) while enhancing the supply of high quality technical education (through new programs, additional classrooms, and infrastructure). The results of the evaluation are summarized below.

Strengthening of 20 Technical Secondary Schools
Evaluator Mathematica Policy Research
Impact or Performance? Impact
Methodology Matched comparison group
Evaluation Period Baseline administrative data comes from 2006-2008, and the baseline survey was done in 2009. Final administrative data comes from 2012, and the final survey round was conducted in 2013. This timeline allows the evaluation to look at educational outcomes for the three years of technical secondary education and labor market outcomes approximately one year after the students should have finished technical secondary school.
Outcomes
  • On average 18 more 10th graders enrolled in technical programs in 2010 in treatment schools than in comparison schools (a 20% increase in enrollment). This trend continued and in 2011 treatment schools had about 36 more students enrolled in technical programs than comparison schools, further increasing to 46 more students in 2012.
  • Final evaluation results found statistically significant impacts of school strengthening on enrollment in technical programs in 2011 and 2012. This impact on enrollment was larger for boys than for girls.
  • Treatment school students had significantly higher PAES global and science scores than comparison school students, likely driven by girls’ PAES scores.
  • No impacts were found on grade progression or graduation rates.
Objective
  • No effect on employment one year after students were schedule to complete general or technical degrees were detected, but there seems to have been a marginal effect on enrollment in technical vocational post-secondary education.
Effect on household income attributable to MCC
  • No effect on income one year after students were scheduled to complete general or technical degrees, due in part to increased enrollment in post-secondary education.

Implementation Findings for the Scholarships for Technical Secondary Schools

The objective of the implementation portion of the evaluation was to document how the components of the sub-activity were designed and assess if they were implemented as planned.

Scholarships for Technical Secondary Schools
Evaluator Mathematica Policy Research
Impact or Performance? Performance
Methodology Mixed methods
Evaluation Period Interviews in 2011, 2014 and 2015; Administrative records reviewed from 2007 to 2015
Outcomes
  • The FOMILENIO scholarships program in strengthened schools provided scholarships to students having an economic need. Scholarships for secondary education for $400 were awarded for the first year and could be renewed up to two years.
  • Students expressed strong satisfaction with FOMILENIO scholarships, but some noted that scholarships did not cover their education expenses.
  • First-year scholarships were distributed in a timely manner by FOMILENIO, but second- and third-year scholarships provided by MINED were not.
  • MINED fulfilled its commitment to fund second- and third-year FOMILENIO scholarships during the post-compact period, but did not fund new first-year scholarships in 2013 or 2014. In 2015, MINED funded a new round of secondary school scholarships in the Northern Zone, but at a level much lower than FOMILENIO.
Objective
  • N/A
Effect on household income attributable to MCC
  • N/A

Impact Findings for the Scholarships for Technical Secondary Schools

For the Scholarships for Technical Secondary Schools, the final impact evaluation found that the scholarships had a positive impact on students’ enrollment, grade completion, and grade progression in secondary school. The final impact findings validate the fundamental logic of the scholarship program, in which modest scholarships can incentivize needy students to enroll in and complete technical degree programs. However, because the scholarships were offered only in schools that also received infrastructure and curriculum improvements, the effects reflect both the offer of the scholarship and the strengthening of the schools.

Scholarships for Technical Secondary School
Evaluator Mathematica Policy Research
Impact or Performance? Impact
Methodology Random assignment
Evaluation Period Scholarships were provided for the 2010 school year and data collection was done in three rounds (July and Aug 2011, July and Aug 2012, and Oct 2013). This timeline allows the evaluation to look at educational outcomes for the three years of technical secondary education and labor market outcomes approximately one year after the students were scheduled to finish technical secondary school.
Outcomes
  •  Applicants who were offered scholarships were 8 percentage points more likely to enroll in secondary school and to earn a secondary degree than students who were not offered scholarships.
  • Males who were offered scholarships were between 14 and 20 percentage points more likely to enroll in and complete secondary school than males not offered scholarships, whereas no statistically significant impacts of scholarships were detected among females for these outcomes.
  • Students who accepted scholarships were 14 percentage points more likely to graduate with a technical degree than those who did not receive one, and 16 percentage points more likely to enroll in a post-secondary technical-vocational education institution
Objective
  • Scholarships had a negative effect on students’ employment. Interpreted in conjunction with the positive effect on post-secondary technical education enrollment suggests that the scholarships, and likely the link with ITCHA, played a role in motivating students to forego immediate entry into the labor force in favor of pursuing a technical post-secondary degree.
Effect on household income attributable to MCC
  • Scholarships in strengthened schools had no impact on income and personal consumption one year after students were scheduled to complete general or technical degrees.

Strengthening of the Chalatenango Technical Institute

For the Strengthening of the Chalatenango Technical Institute component (ITCHA), the evaluation took the form of a mixed methods performance evaluation and looked at ITCHA’s transformation, the role of key players involved in the process, the barriers and facilitators to transforming the institute, and students’ educational and labor market outcomes with regards to enrollment, graduation, and employment to the extent possible. The results are summarized below, which are presented as suggestive effects, but not conclusive.

Strengthening of the Chalatenango Technical Institute
Evaluator Mathematica Policy Research
Impact or Performance? Performance
Methodology  Mixed methods
Evaluation Period 2011-2015
Outcomes
  • New MEGATEC programs blended theory and practice. Introducing the MEGATEC programs included adopting a new approach to student assessment. Teachers and students expressed some difficulties with grading under the competency-based approach.
  • Challenges to implementing the new degree programs included dealing with crowded classrooms, resolving discrepancies in academic standards, and assuring students about the new programs’ legitimacy.
  • Teachers noted improvements in the quality of education but would have liked more guidance, less redundancy in modules, and more realistic goals with respect to the competencies students were expected to demonstrate at the end of teaching modules.
  • Enrollment at ITCHA more than doubled from 309 students in 2008 to 663 in 2011, exceeding the target of 540 students. However, enrollment dipped substantially from 2012 to 2014, and has not been able to reach its target of 540 since.
  • Scholarships likely played a pivotal role in students’ enrollment at ITCHA, reflected by three-fourths of interviewed ITCHA students reporting that they would not have been able to attend without a scholarship.
  • ITCHA students reported grade point averages (GPAs) of around 8.0 out of 10 during their two years at ITCHA.
  • Graduation rates for both cohorts were 85% and above, surpassing the compact target.
Objective
  • Internship arrangements for MEGATEC programs were weaker than originally planned.
  • Across all degree programs, employment rates at follow-up were 52 to 62% in the 2011-2012 and 2012-2013 cohorts, respectively, which are below the compact target of 70% employment one year after ITCHA graduation. While 7% of all interviewed ITCHA students across both cohorts reported not working at follow-up because they were engaged in university studies, it is likely that economic factors, particularly limited demand for these technical skills in the region, was the underlying reason for these employment rates.
  • Employment rates were highest among civil engineering students and lowest among tourism and computing students at 69% and 55%, respectively, among 2012-2013 cohort students.
  • Despite no meaningful differences in GPA or graduation rates across all degree programs, gender imbalances were found between males and females in terms of employment and income. Females in both ITCHA cohorts reported employment rates 13 to 16 percentage points lower than males, and employed males made at least $55 more, on average, than employed females on a monthly basis.
Effect on household income attributable to MCC
  • N/A

Lessons Learned

MCC has identified a set of lessons learned from the evaluations of the El Salvador Formal Technical Sub-Activity.

  • In order to be sustainable, projects should integrate new activities into the already existing system and improve the system where necessary. Teacher training took place by consultants paid with compact funds during implementation, but the on-going teacher training program was ad hoc and teachers and students viewed it be insufficient. This means that new teachers receive informal training from fellow teachers, which puts the reputation of the MEGATEC degrees at risk. It’s important to analyze whether or not a one-off training of teachers represents the highest return or if a systematic improvement to teacher training would provide higher returns. Additionally, the management of the consultants was conducted largely without the Ministry of Education. This can limit ownership and the potential for increasing Ministry capacity. However, the Ministry would have needed additional human resources to do this effectively.
  • Involvement of the private sector is an important aspect of providing quality education and training. Results from the ITCHA evaluation suggest that there may not be enough local labor market demand to absorb secondary school and ITCHA graduates, particularly graduates with alternative tourism degrees. Additionally, the ITCHA had difficulties finding adequate internships for their students. This highlights the importance of having a mechanism to ensure the provision of training is closely linked to the skills demanded in the local or regional labor market.
  • Sustainability mechanisms need to be incorporated into project design from the beginning. Even though many of the components of the sub-activity have been found to have short-term impacts, there is concern about the sustainability of some of the investments, and therefore the sustainability of impacts on outcomes. For example, the funding for scholarships for both secondary schools and ITCHA ended with the compact and it is now the responsibility of the Ministry of Education (MINED) to continue providing them or find another provider. Without continuation of the scholarship program, transportation and other indirect costs will remain significant barriers to student enrollment, particularly for boys. Another example is that while the project succeeded in bringing in new types of teachers with relevant practical experience – after the project, in some cases, they were replaced by more traditional teachers. Demonstrating a new model or approach through exceptions to rules can be a viable short term strategy, but they should be linked to longer term reform efforts to build on the demonstration effect.
  • Project design determines what can be evaluated. An evaluation cannot establish the differential impact of individual components of a package of interventions unless the project is designed to test different components. To do so, different pieces of the package would have to be provided to different groups of beneficiaries. If a package is provided in its entirety, the package will be evaluated and the evaluator will not be able to disentangle the impacts of individual components. This has been illustrated through the evaluation of the Technical Secondary School Strengthening. The evaluation cannot tell us if the increased enrollment is coming from improved infrastructure, better trained teachers, new curriculum, or scholarships. If, for example, scholarships had been made available to a sub-set of students in the comparison schools, the evaluation would have been able to more easily tease out the impact of the school strengthening from the impact of the scholarship. However, the desire and ability to perform such an evaluation depends on the program logic, budget, and the complexity of implementation.
  • Random assignment can be a fair and transparent mechanism to distribute finite resources and allow for a rigorous impact evaluation. In El Salvador, random assignment was seen as a fair way to distribute scholarships when qualified demand exceeded supply. Scholarships were offered for multiple years and the random selection was only conducted in the year where there were more qualified applicants who met the minimum criteria than scholarships available. By randomly selecting scholarship recipients, the applicants who were not offered scholarships were similar to those offered scholarships and thus created a strong control group to measure impact. The random assignment method (a computerized lottery conducted publicly) was transparent and it was clear that each eligible applicant had an equal chance to receive the scholarship.
  • Projects and evaluations should invest in improving the quality and availability of administrative data. Initially, the plan was that much of the data used for these evaluations would come from administrative records from MINED. However, within MINED, school-level and student-level data come from two different departments and they are not consistent with each other. The two data sources use different definitions for counting enrollment, drop-outs from one school are not tracked to see if they enroll in a different school, and some records have been found incomplete. For this final evaluation, MCC used a third party data collector to gather data directly from schools. MCC did not design the project or the evaluation to invest in improving data quality or availability and therefore, the MINED data has significant weaknesses which limit the quality of decisions that can be made based on those data. The quality of data and the use of data within MINED could be improved to help manage El Salvador’s education sector and improve the accuracy of evaluation analyses.

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets in the Formal Technical Education Sub-Activity.
  • 2. Post-secondary scholarships were also provided, but were not subject to a specific evaluation. The result of those scholarships is part of the analysis covered by the evaluation of the Strengthening of the Chalatenango Technical Institute.