The MCC compact with Namibia was a five-year investment (2009-2014) of $304.5 million. The $8.2 million Vocational Training Grant Fund (VTGF) sub-activity is the subject of an independent impact evaluation summarized here.
- VTGF grants funded scholarships to disadvantaged applicants in order to increase their opportunities for paid employment or self-employment, and higher earnings and income.
- The self-reported training completion rate was 46 percent in the treatment group compared to 17 percent in the control group, a statistically significant impact of 29 percentage points.
- The large positive impacts of the offer of VTGF funding on training participation did not translate into positive impacts on employment or earnings about one year after the end of training.
- Two key lessons learned are that there needs to be a credible approach to diagnosing skills gaps in the labor market and that vocational training interventions need to be aligned with this diagnosed demand.
This evaluation is complete and there are no planned next steps.
The MCC compact with Namibia was a five-year investment (2009-2014) of $304.5 million in three projects: Education, Agriculture, and Tourism. The Education project sought to improve the quality of the workforce in Namibia by enhancing the equity and effectiveness of basic, vocational, and tertiary education. It included six major activities: Improving the Quality of General Education, Vocational Education and Skills Training, Improving Access to and Management of Textbooks, Investment in Regional Study and Resource Centers, Expanding and Improving Access to Tertiary Finance, and Cross-Project Support. The Vocational and Skills Training activity consisted of three sub-activities: (i) establishment of a National Training Fund (NTF); (ii) competitive grants for high priority vocational skills training through the Vocational Training Grant Fund (VTGF); and (iii) expansion and improvement of Community Skills Development Centers and training of the Community Skills Development Foundation’s management staff. The $8.2 million VTGF component is the subject of an independent impact evaluation released by MCC in January 2018, the results of which are summarized here; the NTF and COSDEC evaluations were released simultaneously. This component represents 2.7 percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.
*These figures are based on MCC obligations as of March 2016.
The VTGF sub-activity provided grants to training providers to fund scholarships for trainees in high-priority skill areas. It was designed, in part, as a pilot for funding vocational training under the National Training Fund (NTF), which involves a broader, system-wide reform of the vocational training sector but has many features similar to those of the VTGF. Identification of training and training providers for grants was intended to follow a framework in which Industry Skills Councils (ISCs) would determine the market demand for skills, after which MCA-Namibia and the Namibia Training Authority (NTA) would solicit applications from training providers to meet the specific needs identified. ISCs were expected to determine high-priority areas for training based on their knowledge of the sector, consultations with industry, and market studies.
The sub-activity also funded two smaller pilots. One focused on the Recognition of Prior Learning (RPL) program, which helps people experienced in a certain vocational skills area but lacking formal training to compile a portfolio of evidence of their work experience and have their skills formally assessed and certified. MCA-Namibia and the NTA worked with employers who agreed to participate in the pilot to identify the employees who would participate in the RPL program. Then a mentor introduced RPL candidates to the process by providing a roadmap, including assignments and assessments that must be completed to earn an RPL certificate, and served as the assessor for the program. The second pilot was the reimbursement of employers for the costs of employer-provided training under the NTF in which employers register with the NTA, pay a levy, and submit training evidence for reimbursement. (See the NTF evaluation for a fuller description of the NTF pilot and overall intervention.)
VTGF targeted the following key outputs in conjunction with the various components of the sub-activity: increased availability of training for disadvantaged groups; improved equipment and infrastructure; and implementation of the RPL and employer-provided training pilots. Short-term outcomes included increased capacity for the NTA to manage service-level agreements with training providers, increased quality of training through the investments in tools and infrastructure among training providers; increased enrollment of disadvantaged groups targeted by grants for training in high-priority skills; and an expanded market for training through the competitive bidding process for grant funds. Ultimately, trainees were expected to complete training in high-priority skill areas, and experience increased employment and earnings, thereby leading to a reduction in poverty. The employer-provided training and RPL pilots also were expected to occur (culminating in reimbursement of employers and assessment of candidates, respectively), and these pilots would produce lessons learned.
This final evaluation focuses on the scholarships provided by the VTGF grants, which were by far the largest component of the VTGF sub-activity.
Acronyms: RPL = Recognition of Prior Learning; NTA = Namibia Training Authority; LCDRS = Levy collection, distribution, and reporting system
There were several key assumptions underlying the VTGF program logic during the design of the investment:
- Training providers are on the road to the NTA registration and Namibia Qualifications Authority (NQA) accreditation.
- Training is of sufficient quality.
- RPL certificate is valued in the same way as a traditional vocational training certificate.
- Training is aligned with the demands of the labor market.
For a more detailed version of the program logic, please refer to page 4 of the Vocational Training Evaluation Design Report, which can be found here.
MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot not reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes. When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.
The following table summarizes performance on output and outcome indicators specific to the evaluated program.
|Indicators||Level||Baseline (2009)||Actual Achieved (09/2014)||Target||Percent Complete|
|Vocational trainees assisted through the MCA-N Vocational Training Grant Fund||Output||0||1,500||1,638||92%|
|Graduates from MCC-supported education activities||Outcome||0||541||1,392||39%|
|Namibia Qualifications Authority (NQA)-accredited and/or Namibia Training Authority (NTA)-registered vocational training providers||Outcome||25||64||60||111%|
Source: Closeout ITT from December 2014, which includes data through the end of the compact, based on reporting from MCA-Namibia, the Namibia Training Authority.
The completion rate for the one output target was 92 percent. The average completion rate of outcome targets is 75 percent and targets were met or exceeded in one of the two outcome indicators. 1
The impact evaluation was designed to answer the following questions:
- To what extent did applicants who were offered the opportunity of training through the VTGF receive more training relative to nonfunded qualified applicants?
- To what extent did the VTGF-funded trainees’ employment outcomes improve relative to outcomes of nonfunded qualified applicants?
- To what extent did VTGF-funded trainees have higher earnings and income relative to nonfunded qualified applicants?
- To what extent did increased earnings result from increased wages while employed versus increased employment?
- Did the effects of the VTGF-funded training vary by trainee characteristics?
- What key characteristics or practices of training providers were associated with stronger impacts on employment and earnings?
The VTGF impact evaluation uses a random assignment design to answer the research questions outlined above. Under this design, eligible applicants to each VTGF-funded training in which the number of applications exceeded the number of available slots were randomly assigned by the training provider either to a group that was offered a VTGF scholarship (treatment group) or to one that was not (control group). Training providers conducted random assignment separately for each VTGF-funded training after the training was funded and they had solicited applications from potential trainees. Therefore, random assignment occurred on a rolling basis from the fourth quarter of 2010 to the third quarter of 2014, as the Millennium Challenge Account-Namibia and the Namibia Training Authority funded additional trainings.
The follow-up analysis sample consisted of 1,250 applicants in the treatment and control groups from 26 VTGF trainings, conducted by 10 training providers. These trainings cover about one-third of all VTGF trainings and about one-half of all VTGF-funded trainees, but are not representative of the full set of trainings or trainees. The included trainings ranged in duration from less than one month to 22 months, with a median of 8 months. The final analysis sample reflects an overall follow-up survey response rate of 69 percent (72 percent in the treatment group and 67 percent in the control group).
MCC and the evaluator planned to conduct the follow-up survey 12 months after the end of each training, providing a common window in which to assess impacts across trainings. However, for a variety of reasons, this period varied from 6 to 28 months in practice, though the average was indeed around 12 months.
|Evaluator||Mathematica Policy Research|
|Impact or Performance?||Impact|
|Evaluation Period||· VTGF grants were awarded throughout the compact period, with the first grants awarded in the fourth quarter of 2010 and the last grant awarded in the third quarter of 2014
· Round 1 qualitative data collected: October-November 2014
· Round 2 qualitative data collected: November-December 2015
· The baseline survey was conducted on a rolling basis within 3 months of training start (from December 2011 to July 2014)
· The follow-up survey was conducted on a rolling basis 12 months after training ended, on average, to allow sufficient time for outcomes of interest to manifest (from March 2014 to April 2016)
|Training Enrollment and Completion||· At follow-up, about 59 percent of the treatment group had enrolled in vocational training since they applied compared to 25 percent of the control group, a statistically significant impact of 34 percentage points.
· The self-reported training completion rate was 46 percent in the treatment group compared to 17 percent in the control group, a statistically significant impact of 29 percentage points.
· Impacts on both enrollment and completion were almost 50 percent larger for female applicants compared to males.
|Employment and Earnings||· The large positive impacts of the offer of VTGF funding on training participation did not translate into positive impacts on employment at the time of the follow-up survey.
· In contrast, the treatment group was more likely to be engaged in further vocational training at follow-up: specifically, 14 percent compared to 9 percent of the control group, a statistically significant impact of 5 percentage points.
· Overall, about 54 percent of the treatment group was productively engaged—employed in a paid job or engaged in vocational training—at follow-up compared to 58 percent of the control group, a small difference that was not statistically significant.
· Consistent with the limited impacts on employment, the VTGF scholarship offer had no significant impacts on applicants’ earnings in the month before the follow-up survey.
· Earnings were generally low: more than half the treatment group reported no earnings, and only about one quarter earned more than N$2,000 per month (about US$160 at the average exchange rate in the follow-up survey period).
· The estimated impacts on the distribution of total individual income (including non-earnings components) were also not statistically significant, though the mean was significantly lower in the treatment group.
|Effect on household income attributable to MCC||· Similar to the findings for individual income, the distribution of mean monthly household income was similar in the treatment and control groups but the mean was significantly lower in the treatment group.|
- Improving the quality and relevance of TVET programming requires a package of interventions which are rooted in the key institutional problems identified in the given context. The Namibia Vocational Education and Skills Training Activity did not fully account for the complexities of reforming both the system which plans for and allocates resources (such as the ISCs) and the system of implementing new modalities of TVET programming. To this end, MCC would benefit from conducting a more thorough assessment of the institutions and stakeholders involved in TVET systems in the early stages of Compact development. This could range from labor market supply/demand assessments to behavior change assessments to a detailed stakeholder mapping, which would inform the detailed design of the project. This would allow the project to invest in the most urgent and proximal steps to achieving TVET reform and provide a foundation for sustained improvements.
- Vocational training interventions need a credible, demand-based approach for identifying skills gaps in the labor market. A key part of the Vocational Training Activity’s program logic involved shifting the Namibian vocational training sector from being supply-driven to being driven by the needs of the labor market. However, the interim qualitative evaluation found that although the VTGF training grants were intended to target high-priority skill areas, the process to determine market demand with the ISCs was not as scientific as planned and had not yet produced rigorous guidelines for prioritizing skill areas. The employment results seem to reflect the persistent mismatch between labor supply and demand and suggest that the National Training Fund and other activities aiming to improve employment outcomes should continue efforts to better-calibrate training programs to the needs of the labor market.
- In project implementation, it is critical to understand how planned project components create synergies to achieve the desired impacts. For example, according to the theory of change, VTGF training would be offered in priority areas and by matching the supply of skills to the demand for skills, trainees would find jobs and experienced higher income than they would have without the training. Functioning ISCs were in many ways a necessary precondition to assure that training was offered in priority sectors and skills domains. However, in practice, training was funded in the absence of ISC input, which weakened or broke the link between the demand for skills and the supply of them. The VTGF results reinforce the need to consider implementation decisions, such as moving forward without rigorous demand information from the ISCs and the labor market, within the context of the likely impact on targeted results.
- Scholarships may not be enough to ensure people enroll in and complete training. Although the enrollment and completion rates of the training group were significantly higher than those for the control group, neither was “high” in an absolute sense. To the extent vocational training is successful in increasing employment and earnings outcomes, the NTA and training providers should seek to increase take-up and completion of training by those eligible for funding. To the extent funding is a concern, training providers may want to assess the extent to which scholarships meet trainees’ total costs, and identify ways to help finance funding gaps. Training providers may also want to assess interests during the application process and try to match them to the trainings offered in an effort to increase trainee commitment to participation and completion.
This evaluation is complete and there are no planned next steps.