Summary

July 2, 2014

Measuring Results of the Mongolia Health Project

In Context

The MCC compact with Mongolia was a five-year investment (2007-2013) of $269 million in five projects: (i) the Energy and Environment Project, (ii) the Health Project, (iii) the North-South Road Project, (iv) the Property Rights Project, and (v) the Vocational Education Project.

On April 27, 2009, the Government of Mongolia notified MCC that it intended to withdraw the rail project from the Compact. In January 2010, approximately $188 million from the rail project was reallocated for the expansion of the Health, Property Rights, and Vocational Education projects, and the addition of the new North-South Road and Energy and Environment projects. This reallocation led to the Health project budget increasing from $17 to $42 million.

The Health Project included three components: strengthening the national capacity for the development and implementation of Non Communicable Diseases and Injuries (NCDI) policy and practice, strengthening of health care delivery, and stimulating the population for lifestyle change. The project sought to strengthen the national program for prevention, early diagnosis, and management on NCDI. The Health Project investments comprised of 16% of the total Compact investment. The $42.0 million allocated to the Health Project is the subject of both the results described here and an independent performance evaluation conducted by the Independent Evaluator and released by MCC in June 2014.

Program Logic

Mongolia’s weak institutions, including the health system, were identified as significant constraints to economic growth and development, particularly given the pressures of the country’s abrupt transition to a market economy, the collapse of financial support from Russia, and the rapid urbanization of what traditionally has been a highly dispersed rural herding society. The Compact sought to release the potential of certain critical interlocking human, institutional, and physical resources central to Mongolia’s efforts to broaden and deepen economic development.

Within the Compact, the Health Project aimed to increase the adoption of behaviors that reduce NCDIs among target populations and improve medical treatment and control of NCDIs. The project sought to address the incidence of NCDIs with the goal of increasing productivity and incomes. After the Rail Project was cancelled in 2009, the Health Project was expanded in scope. The 2010 amendment expanded the project target population, added an additional intervention of HPV vaccination, and expanded the heart disease and stroke prevention and control programs.

The overall expected result of these changes was an increase in productive years due to reduced incidence of NCDI and savings due to earlier identification of NCDI and thus a reduction in cost of treatment.

According to the M&E Plan, there were several key assumptions underlying the program logic during the design of the investment:

  • Improved system for NCDI prevention will help hospitals and medical centers to detect and treat NCDI effectively.
  • Adoption of healthy lifestyles and preventive measures to avoid NCDI causes will reduce incidence and extend healthy life expectancy of the labor force.
  • Improvement on NCDI early detection activity will reduce the NCDI treatment cost and increase recovery rate.
  • The Government of Mongolia will continue funding the maintenance of key equipment and infrastructure as well as staffing and training of required health professionals.

Measuring Results

MCC uses multiple sources to measure results, including both monitoring and evaluation data.

Monitoring data is generated and used during compact implementation. Monitoring data is typically generated by the program implementers, and specifically covers the program participants who received treatment through the compact.

The following tables summarize performance on outcome and output indicators specific to the evaluated activities1:

Monitoring Indicators Tracked During Implementation of the Productive Development Project
Indicators Level Actual Achieved Target Percent Complete
Prevalence of reduced sodium intake (%) Outcome No data 10.55  
Prevalence of high blood sugar (%) Outcome 8.2 8.2 100%
Prevalence of high blood pressure (%) Outcome 36.2 44 1,400%
Exposure to second-hand smoke (%) Outcome 25.5 32.6 337%
Screening for diabetes (%) Outcome 49.7 66 75%
Screening for hypertension (%) Outcome 58.3 66 88%
Vaccinations against cervical cancer (%) Outcome 6.5 10 65%
At high risk for Non-Communicable Diseases (NCDs) (%) Outcome 36.9 25 -750%
Working population’s awareness of Non-Communicable Disease (NCD) prevention (%) Outcome 98.1 32 *
Cervical cancer cases detected early (%) Outcome 52.6 46.8 169%
Primary healthcare facilities with Non-Communicable Disease (NCD) services (%) Outcome 90.2 70 134%
Screening for cervical cancer (%) Outcome 60.8 26.9 519%
National budget allocated to Non-Communicable Diseases (NCDs) (US Dollars) Output 7,570,000 1,000,000 1,103%
Local government spending towards Non-Communicable Diseases (NCDs) (US Dollars) Output No data    
Grantees for establishing smoke free locations (number) Output 68 40 170%
Health staff trained (number) Output 15,604 5,000 320%
School teachers trained (number) Output 565 565 100%
Staff trained on stroke response (number) Output 15 15 100%
Amount of civil society grants (US Dollars) Output 2,424,099 2,300,000 105%
Grants to workplaces for Non-Communicable Disease and Injury (NCDI) prevention (number) Output 61 50 122%
Non-medical staff trained in response to traffic accidents (number) Output 508 400 127%
Hospitals that treat cervical cancer (%) Output 83.3 100 81%
Health education materials disseminated (number) Output 1,412,055 1,000,000 141%
Facilities with health education materials available (%) Output 91 95 95%

(*= Due to change in definition to "working population" the baseline was recalculated after the final ITT was approved. The new baseline should be 95.8%. Given the inapplicable baseline and target, the percent complete was not calculated.)

For six of the 12 outcome indicators, the target was achieved. For one there was no data, for another the baseline and targets were no longer applicable, and the other four targets were not achieved. For nine of the 12 output indicators, the target was achieved. For one there was no data, and for two the target was not achieved.

The average completion rate of outcome and output targets is 219%. This figure does not take into account the three indicators for which percent complete calculations were not applicable or data was not available. In 16 of the 24 indicators, targets were met or exceeded.

However, the monitoring indicator data collected by the project is limited; it does not permit qualitative assessment of the activities nor a comprehensive assessment of the project’s activities achievements and lessons learned. This is why MCC also invests in independent evaluations.

Evaluation Questions

The independent performance evaluation of the Health Project aimed to assess whether or not:

  • Activities were implemented according to the original plans or undertook new activities;
  • Activities were implemented partially or fully;
  • There were demonstrated changes in outcomes, including knowledge, attitudes and practices; and
  • Project interventions were sustainable.

Evaluation Results

To comprehensively evaluate the MCA Mongolia Health Project, the Independent Evaluator compared various survey data, reviewed project documents, and conducted interviews with project stakeholders.

 
Evaluator Pim de Graaf, Independent Evaluation Consultant
Evaluation Type Performance
Methodology Pre-post survey data comparison; ex-post interviews
Exposure Period Between 24 and 30 months for each intervention.
Immediate and Intermediate Outcomes The evaluator reached the following conclusions (more can be found in Chapter 11 of the full report):
  1. The Health Project has implemented all the activities of the amended Compact, and more than that. In spite of serious obstacles, most activities were well planned and implemented. Exception to be made for the planning of the Stroke and AMI Units.
  2. The Health Project has strengthened the health system rather than weakened it, although there are areas for improvement.
  3. Beyond NCDIs, the Health Project has strengthened a culture of quality and thoroughness of planning and analysis in the Health System. It also has stimulated a culture of efficiency and accountability in the health sector.
  4. The period of actual implementation of the Health Project was between 24 and 30 months. This is far too short for a complex program that is so deeply embedded in the health system. Many components of the Health Project are half-finished, like training and the establishment of smooth running screening programs.
  5. In terms of sustainability, the Health Project missed a few opportunities, like leaving behind a costing plan for further investment and activities on NCDI’s and a repository of relevant documents. Bits and pieces have been left behind but no complete repository.
Ultimate Impact Assessing the achievement of the ultimate goal of the Health Project, increase of life expectancy of the Mongolian population, is not possible within the time frame of the project or immediately afterwards.

Lessons Learned

There were several key lessons learned from this evaluation for MCC and future partner countries to consider when designing and implementing health projects and evaluations:

  1. Set realistic time horizons. The period of actual implementation of the Health Project was only a few years due to complications in start-up and getting the project into implementation mode, including delays in the Government of Mongolia’s Ministry of Health implementing clinical guidelines necessary for the screening campaign and the major reallocation to the project in year two. This is far too short for a complex program that is so deeply embedded in the health system. Potential delays in implementation should be considered during compact development and an appropriate buffer should be built into the project and evaluation timeline. Due to the short Compact time frame, the training may have been incomplete or insufficient to ensure sustainability of the screening programs. This may have been mitigated with more preparation during the compact development phase. Regardless, with the unexpectedly large reallocation from the Rail Project in 2009, MCC may learn to build stronger plans for course corrections or scale up into compact development.
  2. Assess and ensure financial sustainability during development phases. In terms of sustainability, the Health Project missed an opportunity to work with the Government of Mongolia on a sustainability plan to ensure appropriate operations and maintenance budget was developed for equipment, and appropriate staffing and training plans were developed for continuation post-Compact. In the future, this should be built into compact implementation and programming.
  3. Consider social and gender inequalities early on during project design. In this project, certain issues were not identified and addressed until midway through project implementation. These included the higher traffic accident risks for males, which ultimately led the project to implement some education campaigns specifically targeting males. Mobility issues that might prevent disabled people from benefitting were also only properly addressed in the final year of the compact through a targeted campaign for disabled populations. A comprehensive examination of such risks and constraints earlier on would have enabled a more thorough response.
  4. Assess coordination challenges with partners; set benchmarks and milestones. There were many unexpected challenges the project encountered, including upgrading the State Hospital N.3’s Stroke and Cardiac units in order to equip them for high-tech diagnostic equipment, the installation of three gas management lines in the units for sustainability purposes, and general coordination of all parties involved in the project activity. In many cases, these activities required coordination across a range of stakeholders, including MCC, MCA, Government of Mongolia, and other donors. In the future, MCC will work to assess what activities require coordination across stakeholders, define a coordination strategy, with benchmarks and milestones that are tracked between stakeholders in order to mitigate completion and results risks dependent on coordination across many stakeholders.
  5. Structure evaluation to ask about sustainability and behavior change. There was a missed opportunity in assessing how well the behavior change component of the project worked and which aspects of the public awareness campaign had the most impact. The scope of the evaluation did not cover this, primarily because the evaluation was completed directly after the project ended rather than after one or more years to observe behavior changes. It would have been helpful for the health community to understand what outreach activities worked well and which did not have as much success. Also, a later evaluation would have permitted a more substantive assessment of the financial sustainability of the project investments which would have been useful for ERR and beneficiary impact analysis.

Next Steps

The evaluation is complete and there are no planned next steps.

Footnotes
  • 1. These tables reflect data from the final approved closeout MCA-Mongolia Indicator Tracking Table (ITT). Please note, the Evaluation Report references data from earlier ITTs which may explain some discrepancy.