Summary of Findings

Measuring Results of the Mongolia I Compact Vocational Education Project

Activity 5: Improvement of Learning Environments Activity Sub-Activity: Equipment Upgrade

View as PDF

The MCC compact with Mongolia was a five-year investment (2008-2013) of $269 million. The $7.6 million equipment upgrade component of the Vocational Education Project is the subject of an independent impact evaluation summarized here.

  • This component of the Vocational Education Project provided upgraded equipment to technical vocational education and training (TVET) schools to increase employment and income among the unemployed and marginally employed by improving technical skills and productivity, and by developing the TVET sector to become more responsive to labor market demands.
  • The evaluation found no impact of exposure to upgraded TVET equipment on various measures of employment and earnings for either men or women.
  • Major Lesson Learned: A detailed understanding of the problem to be addressed is necessary (for example, a shortage of technical skills in the labor market), leading to an investment that addresses its identified root causes (for example, weak links between training and the private sector, low teacher capacity, lack of financing, inadequate equipment, etc.). Moreover, even in cases where problem diagnosis is exhaustive, documenting the identified problem and a detailed theory of change is a prerequisite to thorough monitoring and evaluation of an investment.
  • This evaluation is complete and there are no planned next steps. A performance evaluation of the remaining activities in the Vocational Education Project is ongoing. The final report is expected in 2019.

In Context

The MCC compact with Mongolia was a five-year investment (2008-2013) of $269 million in 5 projects:  the Energy and Environment Project, the Health Project, the North-South Road Project, the Property Rights Project and the Vocational Education Project.

 

The Vocational Education Project included 5 major activities: the Reforms to TVET Policy and Operational Framework Activity, Creation of Skills Standards and Competency System Activity, Career Guidance and Labor Market Information Systems Development Activity, the Competency-Based Training System Activity, , and the Improvement of Learning Environments Activity.

 

The “Improvement of Learning Environments Activity” consisted of two main components: the provision of upgraded equipment to TVET schools and the construction and rehabilitation of TVET schools. The $7.6 million equipment upgrade component is the subject of an independent impact evaluation released by MCC in August 2018 the results of which are summarized here.  This component represents 3% percent of the total compact. Other components of the compact are the subject of released or forthcoming independent evaluations.

Pie chart outlining the spending in Mongolia Compact I

 

Program Logic

As the Mongolian economy transitioned to a market economy after the fall of the Soviet Union, its technical vocational education and training (TVET) system remained outdated and dysfunctional. The technical content of the trade and occupational courses no longer applied to increasingly sophisticated economy sectors. TVET Learning programs focused heavily on theory, and graduates had a reputation of being ill-equipped for the modern workplace.

In response to a growing economy‘s strong demand for skilled workers, the Vocational Education Project was designed to upgrade technical education and training (the component evaluated here contributed by investments in upgraded equipment). This would increase employment and income among the unemployed and marginally employed by improving technical skills and productivity, and by developing the TVET sector to become more responsive to labor market demands.

By the end of the compact, nearly 12,000 students had graduated from MCC-supported educational facilities, 54 technology labs had been installed and upgraded, and 106 practical training sites had been modernized.

Additional funds were made available after the Rail Project was withdrawn from the compact.

Flow chart describing the program logic

There were several key assumptions underlying the equipment upgrade component of the Vocational Education Project program logic during the design of the investment:

  • Provision of up-to-date equipment, technology and tools and upgrading will enable TVET schools to train graduates who can meet labor market demand after graduation in the long run.
  • TVET schools graduates will support the supply of technical labor force and contribute to the construction and production industry.
  • Imported work force in mining and construction sectors will be decreased.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell one whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC obtains independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC undertakes performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

 

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators Level Baseline (2009) Actual Achieved

(Sep-2013)

Target Percent Complete
Educational facilities constructed or rehabilitated Output  

0

18 18 100
Practical training sites upgraded (equipped) Output  

0

106 90 118
Value of construction, rehabilitation, and/or equipping contracts disbursed Process  

0

27,490,984 15,200,000 181
Percent disbursed of educational facility construction, rehabilitation, and equipping contracts Process 0 97.6 80 122

Source:  Closeout ITT from September 2013, which includes data through the end of the compact, based on reporting from the MCA Mongolia.

 

The average completion rate of output targets is 109 percent and targets were met or exceeded in 2 of the 2 output indicators.  No outcome indicators were tracked as part of the monitoring of the project. 1

 

Evaluation Question

The evaluation was designed to answer the question: What is the effect of learning and training with upgraded equipment on graduates’ employability and wages?

 

A cost-benefit analysis was not completed for this component (equipment upgrading) of the project since (1) the addition of this component would, under the most conservative assumptions, result in a revised ERR approximately equal to the applicable hurdle rate, and (2) the original ERR was judged to have been a conservative estimate. 2 More detail on the Evaluation Design can be found in the Evaluation Design Report here: https://data.mcc.gov/evaluations/index.php/catalog/82/download/564

 

Evaluation Results

While all aspects of the Vocational Education Project were considered for rigorous impact evaluation, the equipment upgrades component was ultimately identified as the only candidate. The results of the evaluation are summarized in the table below. The remaining components of the Vocational Education Project are evaluated as part of a forthcoming performance evaluation.

A rigorous evaluation was possible for two reasons: 1) TVET schools agreed to randomly select from a pool of qualified applicants to over-subscribed trades and 2) the roll-out of the equipment upgrades over time meant that students in later cohorts were exposed to the upgraded equipment for a longer duration. While the final results show no impact on employment or earnings, they are not definitive. Due to the evaluation design, it is not possible to remove the influence of labor market shocks. For example, it is possible students in certain cohorts were differentially affected by downturns in the labor market.

 
Evaluator             Innovations for Poverty Action
Evaluation Type Impact Evaluation
Methodology    Difference-in-Differences
Evaluation Period Implementation: 2010-2013

Timing of Data Collection: Each cohort participated in a baseline study as part of admission. A follow-up survey was completed after graduation. A short tracking survey was completed annually for earlier cohorts.

Exposure Period: 9 – 24 months

Outcomes The evaluation found no impact on intermediate outcomes such as trade-related knowledge or work intensity.
Objective-level Outcomes The evaluation found no impact of exposure to upgraded TVET equipment on various measures of employment and earnings for either men or women.
Effect on household income attributable to MCC N/A

 

Lessons Learned

  • Understanding the linkage between equipment/resources and expected outcomes: Improving employment outcomes and household incomes may require interventions beyond the provision of technical equipment alone.
  • Problem Diagnosis as a Precursor to Effective Program Design: A detailed understanding of the problem to be addressed is necessary (for example, a shortage of technical skills in the labor market), leading to an investment that addresses its identified root causes (for example, weak links between training and the private sector, low teacher capacity, lack of financing, inadequate equipment, etc.). Moreover, even in cases where problem diagnosis is exhaustive, documenting the identified problem and a detailed theory of change is a prerequisite to thorough monitoring and evaluation of an investment.
    • In developing subsequent compacts, MCC has implemented better problem diagnosis processes, including the Constraints Analysis, which identifies the binding constraints to economic growth, and detailed root cause analysis. In the case of the TVET sector, a root cause analysis should include a comprehensive institutional assessment, a credible approach to identifying skills gaps in the labor market, and an understanding of the current/potential role of the private sector in TVET provision. Evaluation results of recent/ongoing TVET projects are pending in Georgia II, Morocco II, and Cote d’Ivoire.
  • Do not finalize evaluation design until program design is adequately complete: In this case, the evaluation began before the project’s investments were finalized. As a result, the evaluation was only able to measure the impact of a sub-component of one of the five activities.
  • Fully define the theory of change: The MCA, MCC and the independent evaluator should coordinate to ensure intermediate outcomes are articulated and measured.
  • Leave room for adaptation in evaluation design: It is important for the independent evaluator to tailor the choice of evaluation methods to their feasibility in the context of the program’s implementation. Moreover, in many cases changes/challenges in program implementation may necessitate changes in evaluation methods.

 

Next Steps

This evaluation is complete and there are no planned next steps. A performance evaluation of the remaining activities in the Vocational Education Project is ongoing. The final report is expected in 2019.

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets for the Vocational Education Project/Improving Learning Environments Activity.
  • 2. MCC Mongolia Implementation Support Team (2009), Investment Memorandum on the Reallocation of Mongolia Compact Funding (August 12), pp. 5-6.