June 2, 2014
Measuring Interim Results of the El Salvador Education and Training Activity
The first MCC compact with El Salvador was a five-year investment (2006-2012) of $460.9 million in three projects: connectivity, human development and productive development. The Compact’s goal was to advance economic growth and poverty reduction in the Northern Zone of El Salvador. The Human Development Project included two major activities – (1) Education and Training, and (2) Community Development – and was equivalent to 21% of the overall Compact investment. The subject of the evaluations summarized here is the $28 million Education and Training Activity, which included three components: (1) Technical Assistance Sub-Activity, (2) Formal Technical Education Sub-Activity, and (3) Non-Formal Skills Development Sub-Activity. The Formal Technical Education Sub-Activity had two major components – (1) Strengthening the Chalatenango Technical Institute (ITCHA), and (2) Strengthening 20 Technical Secondary Schools. Both of those components included the provision of scholarships. The Education and Training Activity was equivalent to 29% of the overall Human Development Project investment and 6% of the total compact.
The Human Development Project was designed to increase knowledge and skills of residents of the Northern Zone of El Salvador through education and skills development programs, and to increase access to basic services (such as water and electricity) and community infrastructure. Specifically, the objective of the Education and Training Activity was to increase the quality and capacity of formal and non-formal vocational programs to enable these programs to absorb and train greater numbers of secondary school students and expand access to more at-risk youth and young adults.
The program logic for the Education and Training Activity assumed that a package of interventions, including improved infrastructure and equipment, scholarships, teacher training, and new curriculum would reduce barriers for students to enroll in 20 selected technical secondary schools and improve quality of education. It was expected that the improved secondary schools would supply the post-secondary Chalatenango Technical Institute (ITCHA) with students who were better prepared for post-secondary education. At the same time, ITCHA underwent significant improvements (in infrastructure and curriculums) to transform it into a Gradual Educational Model of Technical and Technological Learning (known as MEGATEC for its initials in Spanish) institution. MEGATEC is an educational model based on the premise that technical education should be tailored to regional economies’ labor market demands and structured to impart key competencies required of professionals in technical fields. Within the MEGATEC model, new curriculums would follow a competency-based methodology for which teachers would be specially hired and trained.
In addition to improvements in formal education, residents of the Northern Zone who had been excluded from formal education services were expected to have increased access to non-formal short-term technical training courses. These activities were expected to result in additional graduates from both formal and non-formal technical education and training programs with improved skills. Although not included in the program originally, the Sustainable Labor Insertion Program (knows as PILAS for its initials in Spanish) was added during implementation as a complement to the compact’s educational investments. PILAS was designed to link trained individuals, including secondary school, ITCHA and non-formal training graduates, to employment opportunities, which included either support starting their own business or employment with an existing enterprise. Together, the expected increase in skilled workers plus labor insertion services were expected to increase employment of participants, which was expected to lead to an increase in overall household income.
The beneficiaries of the Education & Training Activity were expected to come from a wide range of residents of the Northern Zone of El Salvador as explained in the table below. The beneficiaries of the Formal Technical Education Sub-Activity were expected to be part of the formal education system and a small group of students were expected to benefit from both the secondary school improvements and the ITCHA components. However, the Non-Formal Skills Sub-Activity was targeted towards youth and adults outside of the formal education system.
|Sub-Activity||Component||Targeted Beneficiaries||Cohort included in Interim Evaluations 1||Offered PILAS if passed the selection stage?|
|Formal Technical Education Sub-Activity||Secondary Schools and scholarships||
||2010 to 2011||Yes|
|Chalatenango Technical Institute (ITCHA) and scholarships||
||2007 to 2011||Yes|
|Non-Formal Skills Development Sub-Activity||
||May 2009 to December 2010||Yes|
There were several key assumptions underlying the Education and Training Activity program logic during the design of the investment:
- Lack of capacity of schools and lack of funds by students were the primary reasons for lower enrollment
- Current educational quality was low and new technical curriculums and short courses would teach skills demanded by employers
- Sufficient job opportunities exist for additional graduates of technical education and training
- The Ministry of Education of El Salvador will continue to maintain infrastructure, continue to provide scholarships, and continue to train teachers after the Compact to ensure sustainability
MCC uses multiple sources to measure results, including monitoring data during Compact implementation, and independent evaluations, which in many cases are continued Post Compact. Monitoring data is typically generated by the implementers, and specifically covers the ‘treatment’ group of participants who received services under the Compact.
The table below includes the monitoring indicators that were tracked during implementation of the Education & Training Activity.
|Indicators||Level||Actual Reported||End of Compact Target||Percent Complete|
|Formal Technical Education – Overall|
|Graduates from MCC-supported education activities||Outcome||4,285||6,795||63%|
|Graduates from MCC-supported education activities (post-secondary)||Outcome||405||n/a||n/a|
|Graduates from MCC-supported education activities (secondary)||Outcome||3,880||n/a||n/a|
|Students participating in MCC-supported education activities||Outcome||30,672||26,927||114%|
|Students participating in MCC-supported education activities (post-secondary)||Outcome||1,641||n/a||n/a|
|Students participating in MCC-supported education activities (secondary)||Outcome||17,135||n/a||n/a|
|Instructors and administrators trained||Output||566||500||113%|
|Instructors trained (female)||Output||182||n/a||n/a|
|Formal Technical Education – Chalatenango Technical Institute|
|Graduation rate of the Chalatenango Technical Institute||Outcome||88%||73%||122%|
|Enrolled students of the Chalatenango Technical Institute (2012)||Outcome||613||540||126%|
|Scholarships granted to students for post-secondary technical education (cumulative since 2008)||Output||921||932||99%|
|Scholarships granted to students for post-secondary technical education (cumulative since 2008) (female)||Output||424||359||118%|
|Chalatenango Technical Institute remodeled and equipped||Output||1||1||100%|
|Formal Technical Education – Technical Secondary Schools|
|Graduation rate of technical secondary schools||Outcome||67%||71%||94%|
|Enrolled students of technical secondary schools (2012)||Outcome||9,720||9,413||117%|
|Scholarships granted to students at the technical secondary schools (cumulative since 2009)||Output||3,409||3,445||99%|
|Scholarships granted to students at the technical secondary schools (cumulative since 2009) (female)||Output||1,842||1,464||126%|
|Technical secondary schools remodeled and equipped||Output||20||20||100%|
|Non-Formal Skills Development|
|Employment of graduates of non-formal training programs||Outcome||2,490||1,875||133%|
|Non-formal participants that complete the training||Outcome||11,345||6,888||165%|
|Non-formal training participants (cumulative)||Output||11,896||8,400||142%|
|Non-formal training participants (cumulative; female)||Output||6,963||n/a||n/a|
The average completion rate of output and outcome targets is 114% and the number of indicators for which targets were set where targets were met or exceeded is 12 of 16. It should be noted that these numbers are not always the same as the evaluation results because in addition to not taking the “without project scenario” into account as described below, the monitoring data comes from different data sources, data collection instruments, and samples of respondents.
Monitoring data is limited in that it cannot tell us what these students would have done in the absence of the MCC-funded services. For example, when schools report that enrollment has increased, we do not know if the students enrolled at higher rates because of scholarships or if they would have enrolled anyway. This is a key motivation for why MCC invests in independent impact evaluations, which estimate a counterfactual – what would have happened in the absence of the investment. For some activities, impact evaluations are not feasible or cost-effective and in those cases, MCC invests in independent performance evaluations. The evaluations for the Education and Training Activity combine the use of impact evaluations and performance evaluations.
|Strengthening of the Chalatenango Technical Institute||Performance||Case study|
|Strengthening of 20 Technical Secondary Schools||Impact||Matched comparison group|
|Scholarships for Technical Secondary School||Impact||Randomized control trial|
|Non-Formal Skills Development||Performance||Pre-Post|
The evaluations of the Education and Training Activity were customized for each component and were designed to answer the following questions:
|Strengthening of the Chalatenango Technical Institute||
|Strengthening of 20 Technical Secondary Schools||
|Scholarships for Technical Secondary School||
|Non-Formal Skills Development||
Interim Evaluation Results
Education and Training Activity Overall
The Education and Training Activity evaluations were not designed to quantitatively examine the overall effects of the combined Activity. This is because the two sub-activities had different beneficiary populations and because more time would be required to assess the joint effect of improvements to the technical secondary schools and the post-secondary institute, as students would need to complete all of those years of schooling. However, the final evaluation for Formal Technical Education Sub-Activity will attempt to assess the impact of secondary school improvements and scholarships on performance at ITCHA. In addition, these interim evaluations were not designed to assess the impact of PILAS because the PILAS intervention was identified and designed late in the program. The final evaluations will assess the potential effects of PILAS on each beneficiary population; however, it will be challenging to attribute changes in outcomes to PILAS due to the process PILAS used to select participants. Each component interim evaluation is discussed in turn.
Strengthening of the Chalatenango Technical Institute
For the Strengthening of the Chalatenango Technical Institute component (ITCHA), the main objective of the interim evaluation was to document the process of ITCHA’s transformation, the role of key players involved in the process, the barriers and facilitators to transforming the institute, and initial student outcomes of enrollment, graduation, and employment to the extent possible. The results are summarized below, which are presented as suggestive effects, but not conclusive.
|Evaluator||Mathematica Policy Research|
Implementation was completed in early 2010
Interim evaluation covers 2010 and 2011 (two school years)
Strengthening of 20 Technical Secondary Schools
For the Strengthening of the 20 Technical Secondary Schools, the main objective of the interim evaluation was to analyze initial student outcomes of enrollment, grade completion, and continuation in secondary school for the first school year after the improvements had been completed. Positive effects were found on 10th grade enrollment in technical programs; however the evaluation is unable to separate the effects of the program from other related compact interventions, in particular scholarships provided to 10th graders. 2 No impacts were found on other outcomes within one year of completion of the intervention. In addition to initial impact estimates, the evaluation provides a description of the administrative data available from the Ministry of Education of El Salvador, highlighting some areas for improvement. Some of the MINED data was being verified at the time of writing the interim evaluation, so results are considered preliminary and will be updated once the verified data are available. The results of the interim evaluation are summarized below.
|Evaluator||Mathematica Policy Research|
|Methodology||Matched comparison group|
Implementation was completed for the 2010 school year
Interim evaluation covers the 2010 school year
Scholarships for Technical Secondary School 3
For the Scholarships for Technical Secondary Schools, the interim impact evaluation found that the scholarships had a positive impact on students’ enrollment, grade completion, and grade progression in secondary school. The interim impacts for the offer of the scholarship (intent to treat) and the acceptance of the scholarship (treatment on the treated) are both summarized below.
|Evaluator||Mathematica Policy Research|
|Methodology||Randomized control trial|
Scholarships provided for the 2010 school year
Interim evaluation covers the 2010 and beginning of the 2011 school years
Non-Formal Skills Development
For the Non-Formal Skills Development component, the main objective of the interim evaluation was to present potential effects of the non-formal training on participants’ employment rates and income. The general finding—that participants were more likely to report being employed and earning higher incomes after participation in the non-formal skills courses—suggests that the program had a positive effect. However, other factors could have also affected participants’ pre-post changes in employment and income, and the evaluation cannot separate the effect of these other factors from the effect of the program. The results of the interim evaluation are summarized below, which are presented as suggestive effects, but not conclusive.
|Evaluator||Mathematica Policy Research|
|Evaluation Period||Follow-up one year after completing the training course|
MCC Lessons Learned
MCC has identified a set of lessons learned from the interim evaluations of the El Salvador Education and Training Activity.
- Projects should integrate new activities into the existing system and improve the system where necessary. Within the ITCHA improvement component, new MEGATEC degree programs were created. However, the new MEGATEC degree programs were not integrated into the existing education system, so at the end of the compact it remained unclear how MEGATEC degrees could be used to partially fulfill university requirements. In addition, teacher training took place by consultants paid with compact funds during implementation, but no MEGATEC teacher training program was formally set-up to continue on-boarding new teachers or for continual professional development of existing teachers. This means that new teachers receive informal training from fellow teachers, which puts the reputation of the MEGATEC degrees at risk. It’s important to analyze whether or not a one-off training of teachers represents the highest return or if a systematic improvement to teacher training would provide higher returns.
- Regular analysis of labor markets is an important aspect of providing quality education and training.
This set of interim evaluations was unable to fully analyze employment outcomes; however, preliminary results from the ITCHA/MEGATEC evaluation suggest that there may not be enough local labor market demand to absorb secondary school and ITCHA graduates, particularly graduates with alternative tourism degrees. This highlights the importance of having a mechanism for conducting labor market analysis on a regular basis and feeding this information into curriculums and other aspects of education and training.
- Sustainability mechanisms need to be incorporated into project design from the beginning. Even though many of the components of the Education and Training Activity have been found to have short-term impacts, there is concern about the sustainability of some of the investments, and therefore the sustainability of impacts on outcomes. For example, the funding for scholarships ended with the compact and it is now the responsibility of the Ministry of Education (MINED) to continue providing them or find another provider. The scholarships did not solve a systematic issue of students not being able to cover transportation and meal costs to attend school. Without continuation of the scholarship program, these constraints will remain significant barriers to students.
- Project design determines what can be evaluated. An evaluation cannot establish the differential impact of individual components of a package of interventions unless the project is designed to test different components. To do so, different pieces of the package would have to be provided to different groups of beneficiaries. If a package is provided in its entirety, the package will be evaluated and the evaluator will not be able to disentangle the impacts of individual components. This has been illustrated through the evaluation of the Technical Secondary School Strengthening. The evaluation cannot tell us if the increased enrollment is coming from improved infrastructure, better trained teachers, new curriculum, or scholarships. If, for example, scholarships had been made available to a sub-set of students in the comparison schools, the evaluation would have been able to more easily tease out the impact of the school strengthening from the impact of the scholarship. Programs that include all of these components may be more expensive than necessary; however, without evaluating the different combinations of components, the most efficient allocation of resources is unknown.
- Project design determines feasibility of evaluation type and methodology.
The design of the non-formal training program was demand-driven. In other words, training was provided in communities that requested training. In addition, MCC was informed from the beginning that there would be no excess demand – there would be no community that requested training that would not be provided with training. Since the program was able to cover every community, it was impossible to identify a strong comparison group up-front. The opposite occurred with the scholarship program, as described in the next lesson.
- Random assignment can be a fair and transparent mechanism to distribute finite resources and allow for a rigorous impact evaluation. In El Salvador, random assignment was seen as a fair way to distribute scholarships when qualified demand exceeded supply. Scholarships were offered for multiple years and the random selection was only conducted in the year where there were more qualified applicants who met the minimum criteria than scholarships available. By randomly offering scholarships, the applicants who were not offered scholarships were similar to those offered scholarships and thus created a strong control group to measure impact. The random assignment method (a computerized lottery conducted publicly) was transparent and it was clear that each eligible applicant had an equal chance to receive the scholarship.
- Projects and evaluations should invest in improving the quality and availability of administrative data Much of the data used for these evaluations comes from administrative records from MINED. Within MINED, school-level and student-level data come from two different departments and they are not consistent with each other. The two data sources use different definitions for counting enrollment, drop-outs from one school are not tracked to see if they enroll in a different school, and some records have been found incomplete. Unfortunately, MCC did not design the project or the evaluation to invest in improving data quality or availability and therefore, the data used for analysis by MINED and by evaluators has significant weaknesses which limit the quality of decisions that can be made based on those data. The quality of data and the use of data within MINED could be improved to help manage El Salvador’s education sector not to mention improve the accuracy of evaluation analyses.
As a result of these lessons learned, MCC has adjusted its operational practices with respect to problem diagnostic, project logic and expected results. Lessons learned have fed into education due diligence, policy considerations, project design options and sustainability strategies. In the design of education activities in compacts under development, attention is given to integrating new activities into existing educational systems; strengthening the systems where necessary; and ensuring that sustainability mechanisms are incorporated early on. To match skills demand with supply and build job relevant skills, labor market analyses are being conducted where appropriate and capacity for continuous labor supply-demand assessments addressed.
Specifically, MCC project operational practices have changed in the following ways:
Policy reform and labor market demand. In response to findings from the first El Salvador compact about weaknesses in the technical and vocational education and training (TVET) system, the second compact will include TVET reform activities. One activity will reform the regulatory environment governing the TVET system. To address the finding that the TVET system was not sufficiently demand-based, the second compact will invest in a continuous assessment process for demand-based TVET. This will involve reforming policies specific to private sector involvement in labor demand assessment, curricula and learning methods and standards for teacher and student certification.
In the Republic of Georgia, the government has prioritized TVET sector reform. MCC‘s investment will be directed to helping transform the TVET system into an industry-engaged, market-led system. This includes policy reform to encourage private-sector delivery and better engagement by all TVET providers (public and private) with industry partners. As part of due diligence, a labor market study was conducted to better understand the current mismatch between the skills provided by the TVET system and the skills needed by industries. There are plans to invest in a labor market observatory with public- and private-sector partners that will regularly provide information on skills demand.
Integrating activities in existing education systems and system strengthening. The first El Salvador compact implemented activities at the school level, but did not engage in much system strengthening or policy work. Institutional capacity and policy gaps emerged during the first El Salvador compact that are will be addressed in the second one, specifically:
- Continuous professional development of teachers
- Student assessment system, including classroom, national, and international testing
- Enhancing collection and use of educational data
- Developing a gender policy and strengthening the capacity of MINED in gender integration in curriculum and classroom methodologies.
In the Georgia second compact, a project has been designed to improve competencies of math and science secondary teachers through the existing teacher training system and includes strengthening the capacity of the responsible agency (Teacher Professional Development Center). Strengthening TPDC will enable it to engage in a broad range of teacher professional development in the future and expand the model applied in the compact to teacher training across the country. Sustainability will be enhanced by a commitment from the Ministry of Education and Science to dedicate increased funding to teacher training over the life of the compact.
MCC is also investing in improving the learning environment (rehabbing schools) working through the responsible agency (Education and Science Infrastructure Development Agency, ESIDA). Because of the absence of an O&M system in the Georgian schools system, measures will be taken to put in place a comprehensive O&M program. This will both strengthen ESIDA and provide it with an O&M approach that can be applied system-wide.
- Investing in existing system and ensuring sustainability from the beginning. In the higher education project in the second Georgia compact, MCC is applying lessons learned about integrating activities into existing systems, strengthening them as needed and addressing sustainability measures early on. In the STEM higher education project, MCC will invest in the capacity of the existing public university system to offer international standard STEM bachelor degrees in partnership with international universities. The international universities, selected through an open and competitive process, will partner with established Georgian public universities to strengthen them to provide accredited STEM bachelor degrees. This will include curriculum development, faculty strengthening and upgrading science and lab facilities. Degree programs developed by partners must demonstrate long-term viability through sustainable operating cost levels. Additionally, sustainability is promoted by the GoG’s commitment to provide $50 million over 20 years to support international universities’ operations.
In addition, as a result of these lessons learned, MCC evaluation practices have changed in the following way:
- Formal review process for evaluations. The Monitoring and Evaluation unit is pilot testing a formal review process that defines critical milestones in the evaluation cycle that require substantive review and clearance by key internal stakeholders. This review process also requires local stakeholder review of key evaluation documents in consultation with the evaluator prior to submission to MCC in order to provide feedback on feasibility of proposed evaluation, as well as technical, and factual accuracy of evaluation documents. The formal review process is intended to ensure that evaluations are designed with stakeholder buy-in, are designed using the program logic, use appropriate methodologies for the timeframe of the expected results, and are flexible enough to adjust to changes in implementation.
- Evaluation risk assessment. An Evaluation Risk Assessment Checklist has been developed and institutionalized by the Monitoring and Evaluation unit. The risk assessment checklist is reviewed by the M&E lead with M&E management. The risk assessment is intended to inform decision making and identify necessary course correction for more timely response to risk identification.
- Development and use of standardized evaluation templates. The Monitoring and Evaluation unit has developed standardized templates in order to provide guidance internally and to independent evaluators on expectations related to evaluation activities and products. These templates are intended to clarify and raise standards for evaluations by influencing the daily work of M&E staff and evaluators.
MCC has additional data collection underway and final evaluations will provide more results and learning about the Education and Training Activity. In particular, the final evaluations will include results from graduates of the improved programs one year after graduating and will assess the impact of the interventions on employment and income.
- Final Evaluation of the Formal Technical Education Sub-Activity (Spring 2015)
- Final Performance Evaluation of Non-Formal Skills Development Sub-Activity (Spring 2015)
- 1. Note that none of the students of the improved secondary schools had completed secondary school and continued their studies at ITCHA as of the interim evaluations.
- 2. Scholarships were provided to students in 17 of the 20 secondary schools.
- 3. Post-secondary scholarships were also provided, but were not subject to a specific evaluation. The result of those scholarships is part of the analysis covered by the evaluation of the Strengthening of the Chalatenango Technical Institute.