The first MCC compact with El Salvador was a five-year investment (2006-2012) of $449.6 million in three projects: Connectivity, Human Development and Productive Development. The subject of the evaluations summarized here is the $26 million Education and Training Activity, which made up 31% of the Human Development Project, 6% of the total compact, and aimed to increase the quality and capacity of formal and non-formal vocational programs to enable them to absorb and train greater numbers of secondary school students and expand access to more at-risk youth and young adults. The Education and Training Activity had two components, and the evaluations were designed as such. Interim results for the Formal Technical Education Sub-Activity showed that enrollment at Chalatenango Technical Institute (ITCHA) more than doubled from 2008 to 2011, retention rate increased, and graduation rates exceeded the target. As for the strengthening of the 20 technical secondary schools, positive effects were found on 10th grade enrollment in technical programs in 2010; however, no impacts were found on grade completion, dropout rates, test scores, or re-enrollment rates within one year of completion of the intervention. Scholarships for technical secondary schools showed a positive impact on students’ enrollment, grade completion, and grade progression in secondary school. Additional data collection is underway for a Final Evaluation of the Formal Technical Education Sub-Activity expected to be released in 2016, which will provide more results on graduates of the improved programs one year after graduating and will assess the impact of the interventions on employment and income. As for the Non-Formal Skills Development Sub-Activity, interim and final evaluations both showed higher employment rates. Interim results showed that participants experienced positive changes in principal income, secondary income, additional income, and total net annual income following training, but by the last two rounds of the survey there were no statistically significant income changes. Nonetheless, when all survey rounds with income data were pooled, on average, total net annual income increased by $298 after completing a course. The evaluation also found that participants in the Sustainable Labor Insertion Program (PILAS) were as likely as non-participants to find employment, and as a result there is no conclusive evidence that PILAS assistance led to increased employment and income over and above non-formal skills courses.
In Context 1
The first MCC compact with El Salvador was a five-year investment (2006-2012) of $449.6 million in three projects: connectivity, human development and productive development. The Compact’s goal was to advance economic growth and poverty reduction in the Northern Zone of El Salvador. The Human Development Project included two major activities – (1) Education and Training, and (2) Community Development – and was equivalent to 19% of the overall Compact investment. The subject of the evaluations summarized here is the $26 million Education and Training Activity, which included three components: (1) Technical Assistance Sub-Activity, (2) Formal Technical Education Sub-Activity, and (3) Non-Formal Skills Development Sub-Activity. The Formal Technical Education Sub-Activity had two major components – (1) Strengthening the Chalatenango Technical Institute (ITCHA), and (2) Strengthening 20 Technical Secondary Schools. Both of those components included the provision of scholarships. The Education and Training Activity was equivalent to 31% of the overall Human Development Project investment and 6% of the total compact.
The Human Development Project was designed to increase knowledge and skills of residents of the Northern Zone of El Salvador through education and skills development programs, and to increase access to basic services (such as water and electricity) and community infrastructure. Specifically, the objective of the Education and Training Activity was to increase the quality and capacity of formal and non-formal vocational programs to enable these programs to absorb and train greater numbers of secondary school students and expand access to more at-risk youth and young adults.
The program logic for the Education and Training Activity assumed that a package of interventions, including improved infrastructure and equipment, scholarships, teacher training, and new curriculum would reduce barriers for students to enroll in 20 selected technical secondary schools and improve quality of education. It was expected that the improved secondary schools would supply the post-secondary Chalatenango Technical Institute (ITCHA) with students who were better prepared for post-secondary education. At the same time, ITCHA underwent significant improvements (in infrastructure and curriculums) to transform it into a Gradual Educational Model of Technical and Technological Learning (known as MEGATEC for its initials in Spanish) institution. MEGATEC is an educational model based on the premise that technical education should be tailored to regional economies’ labor market demands and structured to impart key competencies required of professionals in technical fields.
Within the MEGATEC model, new curriculums would follow a competency-based methodology for which teachers would be specially hired and trained.
In addition to improvements in formal education, residents of the Northern Zone who had been excluded from formal education services were expected to have increased access to non-formal short-term technical training courses. These activities were expected to result in additional graduates from both formal and non-formal technical education and training programs with improved skills. Although not included in the program originally, the Sustainable Labor Insertion Program (knows as PILAS for its initials in Spanish) was added during implementation as a complement to the compact’s educational investments. PILAS was designed to link trained individuals, including secondary school, ITCHA and non-formal training graduates, to employment opportunities, which included either support starting their own business or employment with an existing enterprise. Together, the expected increase in skilled workers plus labor insertion services were expected to increase employment of participants, which was expected to lead to an increase in overall household income.
The beneficiaries of the Education & Training Activity were expected to come from a wide range of residents of the Northern Zone of El Salvador as explained in the table below. The beneficiaries of the Formal Technical Education Sub-Activity were expected to be part of the formal education system and a small group of students were expected to benefit from both the secondary school improvements and the ITCHA components. However, the Non-Formal Skills Sub-Activity was targeted towards youth and adults outside of the formal education system.
|Sub-Activity||Component||Targeted Beneficiaries||Cohort included in Interim Evaluation 2||Cohort included in Final Evaluations||Offered PILAS if passed the selection stage?|
|• Transition 9th graders
to 10th graders (from
primary to secondary
• 10th, 11th and 12th
enrolled in the 20
|2010 to 2011||To be included once final evaluation is complete||Yes|
|• Transition 12th grade
students to postsecondary
• Students enrolled at
ITCHA for postsecondary
|2007 to 2011||To be included once final evaluation is complete||Yes|
|• Youth and adults who
were unable or
unlikely to seek
formal education, in
heads of household;
active women and
men (between 17 and
35) regardless of
men and women
between 17 and 24
who completed at
least 9th grade; and
men and women with
|May 2009 to December 2010||January 2011 to March 2012||Yes|
There were several key assumptions underlying the Education and Training Activity program logic during the design of the investment:
• Lack of capacity of schools and lack of funds by students were the primary reasons for lower enrollment
• Current educational quality was low and new technical curriculums and short courses would teach skills demanded by employers
• Sufficient job opportunities exist for additional graduates of technical education and training
• The Ministry of Education of El Salvador will continue to maintain infrastructure, continue to provide scholarships, and continue to train teachers after the Compact to ensure sustainability
MCC uses multiple sources to measure results, including monitoring data during Compact implementation, and independent evaluations, which in many cases are continued Post Compact. Monitoring data is typically generated by the implementers, and specifically covers the ‘treatment’ group of participants who received services under the Compact.
The table below includes the monitoring indicators that were tracked during implementation of the Education & Training Activity.
|Indicators||Level||Actual Reported||End of Compact Total||Percent Complete|
|Formal Technical Education – Overall|
|Graduates from MCC-supported education activities||Outcome||4,285||6,785||63%|
|Graduates from MCC-supported education activities (post-secondary)||Outcome||405||N/A||N/A|
|Graduates from MCC-supported education activities (secondary)||Outcome||3,880||N/A||N/A|
|Students participating in MCC-supported education activities||Outcome||30,672||26,972||114%|
|Students participating in MCC-supported education activities (post-secondary)||Outcome||1,641||N/A||N/A|
|Students participating in MCC-supported education activities (secondary)||Outcome||17,135||N/A||N/A|
|Instructors and administrators trained||Output||566||500||113%|
|Instructors trained (female)||Output||182||N/A||N/A|
|Formal Technical Education – Chalatenango Technical Institute|
|Graduation rate of the Chalatenango Technical Institute||Outcome||88%||73%||122%|
|Enrolled students of the Chalatenango Technical Institute (2012)||Outcome||613||540||126%|
|Scholarships granted to students for postsecondary technical education (cumulative since 2008)||Output||921||932||99%|
|Scholarships granted to students for postsecondary
technical education (cumulative since 2008) (female)
|Chalatenango Technical Institute remodeled and equipped||Output||1||1||100%|
|Formal Technical Education – Technical Secondary Schools|
|Graduation rate of technical secondary schools||Outcome||67%||71%||94%|
|Enrolled students of technical secondary schools (2012)||Outcome||9,720||9,413||117%|
|Scholarships granted to students at the technical secondary schools (cumulative since 2009)||Output||3,409||3,445||99%|
|Scholarships granted to students at the technical secondary schools (cumulative since 2009) (female)||Output||1,842||1,464||126%|
|Technical secondary schools remodeled and equipped||Output||20||20||100%|
|Non-Formal Skills Development|
|Employment of graduates of non-formal training programs||Outcome||2,490||1,875||133%|
|Non-formal participants that complete the training||Outcome||11,345||6,888||165%|
|Non-formal training participants (cumulative)||Output||11,896||8,400||142%|
|Non-formal training participants (cumulative; female)||Output||6,963||N/A||N/A|
The average completion rate of output and outcome targets is 114% and the number of indicators for which targets were set where targets were met or exceeded is 12 of 16. It should be noted that these numbers are not always the same as the evaluation results because in addition to not taking the “without project scenario” into account as described below, the monitoring data comes from different data sources, data collection instruments, and samples of respondents.
Monitoring data is limited in that it cannot tell us what these students would have done in the absence of the MCC-funded services. For example, when schools report that enrollment has increased, we do not know if the students enrolled at higher rates because of scholarships or if they would have enrolled anyway. This is a key motivation for why MCC invests in independent impact evaluations, which estimate a counterfactual – what would have happened in the absence of the investment. For some activities, impact evaluations are not feasible or cost-effective and in those cases, MCC invests in independent performance evaluations. The evaluations for the Education and Training Activity combine the use of impact evaluations and performance evaluations.
Summary of Education and Training Activity Interim Evaluations
|Summary of Education and Training Activity Interim Evaluations|
|Strengthening of the Chalatenango Technical Institute||Performance||Case study|
|Strengthening of 20 Technical Secondary Schools||Impact||Matched comparison group|
|Scholarships for Technical Secondary School||Impact||Randomized control trial|
|Non-Formal Skills Development||Performance||Pre-Post|
The evaluations of the Education and Training Activity were customized for each component and were designed to answer the following questions:
|Strengthening of the Chalatenango Technical Institute||– Did implementation of the ITCHA strengthening activity proceed as planned?
– Did student enrollment and graduation meet expectations?
– Did the new degree programs prepare students for employment and university classes?
|Strengthening of 20 Technical Secondary Schools||– What is the impact of strengthening 20 technical secondary schools on students’ educational and labor market outcomes, including secondary school enrollment, grade completion, graduation, employment, and income?|
|Scholarships for Technical Secondary School||– What is the impact of the scholarships on recipients’ educational and labor market outcomes?|
|Non-Formal Skills Development||– What were course participants’ demographic and socioeconomic characteristics?
– How was the Sub-Activity planned and implemented? What were implementation challenges and facilitators?
– How did participants’ labor market outcomes and income change from before beginning non-formal skills courses to approximately one year after completing the courses?
Interim Evaluation Results
Education and Training Activity Overall
The Education and Training Activity evaluations were not designed to quantitatively examine the overall effects of the combined Activity. This is because the two sub-activities had different beneficiary populations and because more time would be required to assess the joint effect of improvements to the technical secondary schools and the post-secondary institute, as students would need to complete all of those years of schooling. However, the final evaluation for Formal Technical Education Sub-Activity will attempt to assess the impact of secondary school improvements and scholarships on performance at ITCHA. In addition, these interim evaluations were not designed to assess the impact of PILAS because the PILAS intervention was identified and designed late in the program. The final evaluations will assess the potential effects of PILAS on each beneficiary population; however, it will be challenging to attribute changes in outcomes to PILAS due to the process PILAS used to select participants. Each component interim evaluation is discussed in turn.
Strengthening of the Chalatenango Technical Institute – Interim Evaluation
For the Strengthening of the Chalatenango Technical Institute component (ITCHA), the main objective of the interim evaluation was to document the process of ITCHA’s transformation, the role of key players involved in the process, the barriers and facilitators to transforming the institute, and initial student outcomes of enrollment, graduation, and employment to the extent possible. The results are summarized below, which are presented as suggestive effects, but not conclusive.
|Strengthening of the Chalatenango Technical Institute|
|Evaluator||Mathematica Policy Research|
|Evaluation Period||Implementation was completed in early 2010
Interim evaluation covers 2010 and 2011 (two school years)
|Enrollment||• Enrollment at ITCHA more than doubled from 309 students in 2008 to 663 in 2011, exceeding the target of 540 students. However, due to space constraints and limited number of instructors, enrollment in 2012 is not likely to meet the original compact target of 1,100.
• ITCHA’s 2011 second-year retention rate was around 96%, which is substantially higher than the 70% retention rate in 2009 prior to ITCHA’s conversion.
|Graduation||• At 88%, the overall graduation rate for the 2010-2011 cohort had already surpassed the intervention’s target of 73% for the 2012 school year.|
|Employment||• Long-term employment prospects for graduates cannot be determined with certainty. In future years, there may not be enough local labor market demand in the area to absorb secondary school and ITCHA graduates, particularly graduates with alternative tourism degrees.|
|Sustainability||• Given teacher turnover, the sustainability of the intervention (the introduction of competency-based curriculum) is questionable as there is no formal MEGATEC training program in place to train newly hired teachers or financial and human resources to conduct said training.
• While there is strong political support for the new MEGATEC, the sustainability of the intervention is further questioned as the amount and method of payment of scholarships under MINED’s administration had not been defined at the time of writing the interim evaluation.
Strengthening of 20 Technical Secondary Schools – Interim Evaluation
For the Strengthening of the 20 Technical Secondary Schools, the main objective of the interim evaluation was to analyze initial student outcomes of enrollment, grade completion, and continuation in secondary school for the first school year after the improvements had been completed. Positive effects were found on 10th grade enrollment in technical programs; however the evaluation is unable to separate the effects of the program from other related compact interventions, in particular scholarships provided to 10th graders. 3 No impacts were found on other outcomes within one year of completion of the intervention. In addition to initial impact estimates, the evaluation provides a description of the administrative data available from the Ministry of Education of El Salvador, highlighting some areas for improvement. Some of the MINED data was being verified at the time of writing the interim evaluation, so results are considered preliminary and will be updated once the verified data are available. The results of the interim evaluation are summarized below.
|Strengthening of 20 Technical Secondary Schools|
|Evaluator||Mathematica Policy Research|
|Methodology||Matched comparison group|
|Evaluation Period||Implementation was completed for the 2010 school year
Interim evaluation covers the 2010 school year
|Enrollment||• On average 18 more 10th graders enrolled in technical programs in 2010 in treatment schools than in comparison schools (a 20% increase in enrollment). This impact is found using school-level data.
• No impact was found on enrollment in 2011 using preliminary student-level data.
|Other Outcomes||• There is no impact on other outcomes of grade completion, dropout rates, test scores or re-enrollment rates.|
Scholarships for Technical Secondary School 4 – Interim Evaluation
For the Scholarships for Technical Secondary Schools, the interim impact evaluation found that the scholarships had a positive impact on students’ enrollment, grade completion, and grade progression in secondary school. The interim impacts for the offer of the scholarship (intent to treat) and the acceptance of the scholarship (treatment on the treated) are both summarized below.
|Scholarships for Technical Secondary School|
|Evaluator||Mathematica Policy Research|
|Methodology||Randomized control trial|
|Evaluation Period||Scholarships provided for the 2010 school year
Interim evaluation covers the 2010 and beginning of the 2011 school years
|Enrollment||• Applicants who were offered scholarships were 8 percentage points more likely to enroll in 10th grade in 2010 than students who were not offered scholarships (95 percent versus 87 percent among students not offered scholarships).
• Males who were offered scholarships were between 14 and 16 percentage points more likely to enroll in 10th grade, complete 10th grade, and enroll in 11th grade than males not offered scholarships, whereas no statistically significant impacts of scholarships were detected among females for these outcomes.
• Students who accepted the scholarship were 11 percentage points more likely to enroll in 10th grade.
|Grade Completion||• Applicants who were offered scholarships were 6 percentage points more likely to complete 10th grade than students who were not offered scholarships (85 percent versus 79 percent among students not offered scholarships).
• Students who accepted the scholarship were 9 percentage points more likely to complete 10th grade.
|Grade Progression||• Applicants who were offered scholarships were 8 percentage points more likely to enroll in 11th grade in 2011 than students who were not offered scholarships (84 percent versus 76 percent among students not offered scholarships).
• Students who accepted the scholarship were 10 percentage points more likely to enroll in 11th grade.
Non-Formal Skills Development – Interim and Final Evaluations
For the Non-Formal Skills Development component, the main objective of the interim and final evaluations was to present changes before and after the program on participants’ employment rates and income. The general finding—that participants were more likely to report being employed and earning higher incomes after participation in the non-formal skills courses—suggests that the program had a positive effect. However, other factors could have also affected participants’ pre-post changes in employment and income, and the evaluation cannot separate the effect of these other factors from the effect of the program. In addition, because the study used retrospective assessment of employment and income before the program, the changes could also reflect the difference in data collection from pre to post. The results of the interim and final evaluations are summarized below, which are presented as suggestive effects, but not conclusive.
|Non-Formal Skills Development|
|Evaluator||Mathematica Policy Research|
|Evaluation Period||Follow-up one year after completing the training course|
|Participant Characteristics||• Final: The sample consisted of 1,160 individuals who completed their first non-formal skills training course between January 2011 and March 2012.
o Individuals in this sample were an average of 30 years old. The sample is predominantly rural.
o Sixty-seven percent of respondents were women and 34 percent of respondents (or approximately half the women) were unemployed women between 17 and 35 years of age at baseline.
o Survey respondents had 2.9 years of work experience at baseline and 8.1 years of education, on average, at the time of the follow-up survey.
|Employment||• Interim: Following their completion of non-formal skills training courses, participants’ employment rates increased by 30 percentage points, with a 15 percentage point increase in self-employment and a 10 percentage point increase in salaried employment.
Final: In the later data collection rounds, which are likely more accurate due to improved survey instruments, employment growth was smaller at 14 percentage points. When all rounds of data are pooled together, employment rates increased by 26 percentage points; however, given the weaknesses in early survey rounds, this is likely an overestimate.
• Interim: Participants who took courses related to food preparation, such as cooking and baking, were more likely to be self-employed than obtain salaried employment following training. In contrast, participants who took courses in bricklaying and residential electrical installations experienced greater increases in salaried employment rates as compared to self-employment rates.
Final: Bakers, cooks, and auto mechanics were more likely to find salaried employment, whereas those who studied tailoring or pastry making were more likely to find self-employment in some rounds. It isn’t clear why this is different from the interim evaluation.
• Interim and Final: Self-employment increased more among women, and salaried employment increased more among men.
• Interim: The least-educated participants experienced the least success in obtaining a job following training.
Final: In later survey rounds, the results were different, as there was no clear relationship between education level and success in obtaining a job.
|Income||• Interim: Participants experienced positive changes in principal income, secondary income, additional income, and total net annual income following training. Increases in primary income were particularly large among the newly employed, especially those who obtained salaried positions after training. In addition, cooking and electrical installation courses are associated with the largest income increases, followed by baking and bricklaying courses.
Final: There were no statistically significant income changes in the last two rounds of the survey. However, in the last four rounds of the survey, total income increased significantly for participants in the tailoring and pastry-making courses, but not for the other courses. When all rounds with income data are pooled, on average, total net annual income increased by $298 after completing a course. However, it is likely that these estimates are biased upward due to the limitations of the survey instrument used in early survey rounds.
• Interim: Men and women experienced similar income increases following training.
Final: In the later rounds, women experienced larger income increases than men following training.
|PILAS||• Final: PILAS participants were as likely as non-participants to find employment. Both PILAS participants and nonparticipants reported increases in total income, but PILAS participants experienced larger income increases in secondary income ($129 for PILAS participants versus $31 for nonparticipants). This is difficult to interpret since PILAS was designed to affect primary income. Therefore, there is no conclusive evidence that PILAS assistance led to increased employment and income over and above non-formal skills courses.|
MCC Lessons Learned
MCC has identified a set of lessons learned from the evaluations of the El Salvador Education and Training Activity.
• Projects should integrate new activities into the existing system and improve the system where necessary. Within the ITCHA improvement component, new MEGATEC degree programs were created. However, the new MEGATEC degree programs were not integrated into the existing education system, so at the end of the compact it remained unclear how MEGATEC degrees could be used to partially fulfill university requirements. In addition, teacher training took place by consultants paid with compact funds during implementation, but no MEGATEC teacher training program was formally set-up to continue on-boarding new teachers or for continual professional development of existing teachers. This means that new teachers receive informal training from fellow teachers, which puts the reputation of the MEGATEC degrees at risk. It’s important to analyze whether or not a one-off training of teachers represents the highest return or if a systematic improvement to teacher training would provide higher returns.
• Regular analysis of labor markets is an important aspect of providing quality education and training. This set of interim evaluations was unable to fully analyze employment outcomes; however, preliminary results from the ITCHA/MEGATEC evaluation suggest that there may not be enough local labor market demand to absorb secondary school and ITCHA graduates, particularly graduates with alternative tourism degrees. This highlights the importance of having a mechanism for conducting labor market analysis on a regular basis and feeding this information into curriculums and other aspects of education and training.
• Sustainability mechanisms need to be incorporated into project design from the beginning. Even though many of the components of the Education and Training Activity have been found to have short-term impacts, there is concern about the sustainability of some of the investments, and therefore the sustainability of impacts on outcomes. For example, the funding for scholarships ended with the compact and it is now the responsibility of the Ministry of Education (MINED) to continue providing them or find another provider. The scholarships did not solve a systematic issue of students not being able to cover transportation and meal costs to attend school. Without continuation of the scholarship program, these constraints will remain significant barriers to students.
• Project design determines what can be evaluated. An evaluation cannot establish the differential impact of individual components of a package of interventions unless the project is designed to test different components. To do so, different pieces of the package would have to be provided to different groups of beneficiaries. If a package is provided in its entirety, the package will be evaluated and the evaluator will not be able to disentangle the impacts of individual components. This has been illustrated through the evaluation of the Technical Secondary School Strengthening. The evaluation cannot tell us if the increased enrollment is coming from improved infrastructure, better trained teachers, new curriculum, or scholarships. If, for example, scholarships had been made available to a sub-set of students in the comparison schools, the evaluation would have been able to more easily tease out the impact of the school strengthening from the impact of the scholarship. Programs that include all of these components may be more expensive than necessary; however, without evaluating the different combinations of components, the most efficient allocation of resources is unknown.
• Project design determines feasibility of evaluation type and methodology. The design of the non-formal training program was demand-driven. In other words, training was provided in communities that requested training (note that this is different from being driven by labor market demand). In addition, MCC was informed from the beginning that there would be no excess demand – there would be no community that requested training that would not be provided with training. Since the program was able to cover every community, it was impossible to identify a strong comparison group up-front. The opposite occurred with the scholarship program, as described in the next lesson.
• Random assignment can be a fair and transparent mechanism to distribute finite resources and allow for a rigorous impact evaluation. In El Salvador, random assignment was seen as a fair way to distribute scholarships when qualified demand exceeded supply. Scholarships were offered for multiple years and the random selection was only conducted in the year where there were more qualified applicants who met the minimum criteria than scholarships available. By randomly offering scholarships, the applicants who were not offered scholarships were similar to those offered scholarships and thus created a strong control group to measure impact. The random assignment method (a computerized lottery conducted publicly) was transparent and it was clear that each eligible applicant had an equal chance to receive the scholarship.
• Projects and evaluations should invest in improving the quality and availability of administrative data. Much of the data used for these evaluations comes from administrative records from MINED. Within MINED, school-level and student-level data come from two different departments and they are not consistent with each other. The two data sources use different definitions for counting enrollment, drop-outs from one school are not tracked to see if they enroll in a different school, and some records have been found incomplete. Unfortunately, MCC did not design the project or the evaluation to invest in improving data quality or availability and therefore, the data used for analysis by MINED and by evaluators has significant weaknesses which limit the quality of decisions that can be made based on those data. The quality of data and the use of data within MINED could be improved to help manage El Salvador’s education sector not to mention improve the accuracy of evaluation analyses.
As a result of these lessons learned, MCC has adjusted its operational practices with respect to problem diagnostic, project logic and expected results. Lessons learned have fed into education due diligence, policy considerations, project design options and sustainability strategies. In the design of education activities in compacts under development, attention is given to integrating new activities into existing educational systems; strengthening the systems where necessary; and ensuring that sustainability mechanisms are incorporated early on. To match skills demand with supply and build job relevant skills, labor market analyses are being conducted where appropriate and capacity for continuous labor supply-demand assessments addressed.
Specifically, MCC project operational practices have changed in the following ways:
• Policy reform and labor market demand. In response to findings from the first El Salvador compact about weaknesses in the technical and vocational education and training (TVET) system, the second compact will include TVET reform activities. One activity will reform the regulatory environment governing the TVET system. To address the finding that the TVET system was not sufficiently demand-based, the second compact will invest in a continuous assessment process for demand-based TVET. This will involve reforming policies specific to private sector involvement in labor demand assessment, curricula and learning methods and standards for teacher and student certification.
In the Republic of Georgia, the government has prioritized TVET sector reform. MCC‘s investment will be directed to helping transform the TVET system into an industry-engaged, market-led system. This includes policy reform to encourage private-sector delivery and better engagement by all TVET providers (public and private) with industry partners. As part of due diligence, a labor market study was conducted to better understand the current mismatch between the skills provided by the TVET system and the skills needed by industries. There are plans to invest in a labor market observatory with public- and private-sector partners that will regularly provide information on skills demand.
• Integrating activities in existing education systems and system strengthening. The first El Salvador compact implemented activities at the school level, but did not engage in much system strengthening or policy work. Institutional capacity and policy gaps emerged during the first El Salvador compact that are will be addressed in the second one, specifically:
o Continuous professional development of teachers
o Student assessment system, including classroom, national, and international testing
o Enhancing collection and use of educational data
o Developing a gender policy and strengthening the capacity of MINED in gender integration in curriculum and classroom methodologies.
In the Georgia second compact, a project has been designed to improve competencies of math and science secondary teachers through the existing teacher training system and includes strengthening the capacity of the responsible agency (Teacher Professional Development Center). Strengthening TPDC will enable it to engage in a broad range of teacher professional development in the future and expand the model applied in the compact to teacher training across the country. Sustainability will be enhanced by a commitment from the Ministry of Education and Science to dedicate increased funding to teacher training over the life of the compact.
MCC is also investing in improving the learning environment (rehabbing schools) working through the responsible agency (Education and Science Infrastructure Development Agency, ESIDA). Because of the absence of an O&M system in the Georgian schools system, measures will be taken to put in place a comprehensive O&M program. This will both strengthen ESIDA and provide it with an O&M approach that can be applied system-wide.
• Investing in existing system and ensuring sustainability from the beginning. In the second compact in El Salvador careful attention has been paid to working with the existing system to strengthen it and ensure sustainability during the course of project implementation. Key institutions in both the quality of education and TVET reform activities have been involved in project design. Conditions precedent have been developed to ensure gradual absorption of project activities throughout the course of compact implementation.
• In the higher education project in the second Georgia compact, MCC is applying lessons learned about integrating activities into existing systems, strengthening them as needed and addressing sustainability measures early on. In the STEM higher education project, MCC will invest in the capacity of the existing public university system to offer international standard STEM bachelor degrees in partnership with international universities. The international universities, selected through an open and competitive process, will partner with established Georgian public universities to strengthen them to provide accredited STEM bachelor degrees. This will include curriculum development, faculty strengthening and upgrading science and lab facilities. Degree programs developed by partners must demonstrate long-term viability through sustainable operating cost levels. Additionally, sustainability is promoted by the GoG’s commitment to provide $50 million over 20 years to support international universities’ operations.
In addition, as a result of these lessons learned, MCC evaluation practices have changed in the following way:
• Formal review process for evaluations. The Monitoring and Evaluation unit is pilot testing a formal review process that defines critical milestones in the evaluation cycle that require substantive review and clearance by key internal stakeholders. This review process also requires local stakeholder review of key evaluation documents in consultation with the evaluator prior to submission to MCC in order to provide feedback on feasibility of proposed evaluation, as well as technical, and factual accuracy of evaluation documents. The formal review process is intended to ensure that evaluations are designed with stakeholder buy-in, are designed using the program logic, use appropriate methodologies for the timeframe of the expected results, and are flexible enough to adjust to changes in implementation.
• Evaluation risk assessment. An Evaluation Risk Assessment Checklist has been developed and institutionalized by the Monitoring and Evaluation unit. The risk assessment checklist is reviewed by the M&E lead with M&E management. The risk assessment is intended to inform decision making and identify necessary course correction for more timely response to risk identification.
• Development and use of standardized evaluation templates. The Monitoring and Evaluation unit has developed standardized templates in order to provide guidance internally and to independent evaluators on expectations related to evaluation activities and products. These templates are intended to clarify and raise standards for evaluations by influencing the daily work of M&E staff and evaluators.
MCC has additional data collection underway and final evaluations will provide more results and learning about the Education and Training Activity. In particular, the final evaluations will include results from graduates of the improved programs one year after graduating and will assess the impact of the interventions on employment and income.
• Final Evaluation of the Formal Technical Education Sub-Activity (Spring 2016)