July 2, 2014
Measuring Results of the Mongolia Health Project
The MCC compact with Mongolia was a five-year investment (2007-2013) of $269 million in five projects: (i) the Energy and Environment Project, (ii) the Health Project, (iii) the North-South Road Project, (iv) the Property Rights Project, and (v) the Vocational Education Project.
On April 27, 2009, the Government of Mongolia notified MCC that it intended to withdraw the rail project from the Compact. In January 2010, approximately $188 million from the rail project was reallocated for the expansion of the Health, Property Rights, and Vocational Education projects, and the addition of the new North-South Road and Energy and Environment projects. This reallocation led to the Health project budget increasing from $17 to $42 million.
The Health Project included three components: strengthening the national capacity for the development and implementation of Non Communicable Diseases and Injuries (NCDI) policy and practice, strengthening of health care delivery, and stimulating the population for lifestyle change. The project sought to strengthen the national program for prevention, early diagnosis, and management on NCDI. The Health Project investments comprised of 16% of the total Compact investment. The $42.0 million allocated to the Health Project is the subject of both the results described here and an independent performance evaluation conducted by the Independent Evaluator and released by MCC in June 2014.
Mongolia’s weak institutions, including the health system, were identified as significant constraints to economic growth and development, particularly given the pressures of the country’s abrupt transition to a market economy, the collapse of financial support from Russia, and the rapid urbanization of what traditionally has been a highly dispersed rural herding society. The Compact sought to release the potential of certain critical interlocking human, institutional, and physical resources central to Mongolia’s efforts to broaden and deepen economic development.
Within the Compact, the Health Project aimed to increase the adoption of behaviors that reduce NCDIs among target populations and improve medical treatment and control of NCDIs. The project sought to address the incidence of NCDIs with the goal of increasing productivity and incomes. After the Rail Project was cancelled in 2009, the Health Project was expanded in scope. The 2010 amendment expanded the project target population, added an additional intervention of HPV vaccination, and expanded the heart disease and stroke prevention and control programs.
The overall expected result of these changes was an increase in productive years due to reduced incidence of NCDI and savings due to earlier identification of NCDI and thus a reduction in cost of treatment.
According to the M&E Plan, there were several key assumptions underlying the program logic during the design of the investment:
- Improved system for NCDI prevention will help hospitals and medical centers to detect and treat NCDI effectively.
- Adoption of healthy lifestyles and preventive measures to avoid NCDI causes will reduce incidence and extend healthy life expectancy of the labor force.
- Improvement on NCDI early detection activity will reduce the NCDI treatment cost and increase recovery rate.
- The Government of Mongolia will continue funding the maintenance of key equipment and infrastructure as well as staffing and training of required health professionals.
MCC uses multiple sources to measure results, including both monitoring and evaluation data.
Monitoring data is generated and used during compact implementation. Monitoring data is typically generated by the program implementers, and specifically covers the program participants who received treatment through the compact.
The following tables summarize performance on outcome and output indicators specific to the evaluated activities1:
|Indicators||Level||Actual Achieved||Target||Percent Complete|
|Prevalence of reduced sodium intake (%)||Outcome||No data||10.55|
|Prevalence of high blood sugar (%)||Outcome||8.2||8.2||100%|
|Prevalence of high blood pressure (%)||Outcome||36.2||44||1,400%|
|Exposure to second-hand smoke (%)||Outcome||25.5||32.6||337%|
|Screening for diabetes (%)||Outcome||49.7||66||75%|
|Screening for hypertension (%)||Outcome||58.3||66||88%|
|Vaccinations against cervical cancer (%)||Outcome||6.5||10||65%|
|At high risk for Non-Communicable Diseases (NCDs) (%)||Outcome||36.9||25||-750%|
|Working population’s awareness of Non-Communicable Disease (NCD) prevention (%)||Outcome||98.1||32||*|
|Cervical cancer cases detected early (%)||Outcome||52.6||46.8||169%|
|Primary healthcare facilities with Non-Communicable Disease (NCD) services (%)||Outcome||90.2||70||134%|
|Screening for cervical cancer (%)||Outcome||60.8||26.9||519%|
|National budget allocated to Non-Communicable Diseases (NCDs) (US Dollars)||Output||7,570,000||1,000,000||1,103%|
|Local government spending towards Non-Communicable Diseases (NCDs) (US Dollars)||Output||No data|
|Grantees for establishing smoke free locations (number)||Output||68||40||170%|
|Health staff trained (number)||Output||15,604||5,000||320%|
|School teachers trained (number)||Output||565||565||100%|
|Staff trained on stroke response (number)||Output||15||15||100%|
|Amount of civil society grants (US Dollars)||Output||2,424,099||2,300,000||105%|
|Grants to workplaces for Non-Communicable Disease and Injury (NCDI) prevention (number)||Output||61||50||122%|
|Non-medical staff trained in response to traffic accidents (number)||Output||508||400||127%|
|Hospitals that treat cervical cancer (%)||Output||83.3||100||81%|
|Health education materials disseminated (number)||Output||1,412,055||1,000,000||141%|
|Facilities with health education materials available (%)||Output||91||95||95%|
(*= Due to change in definition to "working population" the baseline was recalculated after the final ITT was approved. The new baseline should be 95.8%. Given the inapplicable baseline and target, the percent complete was not calculated.)
For six of the 12 outcome indicators, the target was achieved. For one there was no data, for another the baseline and targets were no longer applicable, and the other four targets were not achieved. For nine of the 12 output indicators, the target was achieved. For one there was no data, and for two the target was not achieved.
The average completion rate of outcome and output targets is 219%. This figure does not take into account the three indicators for which percent complete calculations were not applicable or data was not available. In 16 of the 24 indicators, targets were met or exceeded.
However, the monitoring indicator data collected by the project is limited; it does not permit qualitative assessment of the activities nor a comprehensive assessment of the project’s activities achievements and lessons learned. This is why MCC also invests in independent evaluations.
The independent performance evaluation of the Health Project aimed to assess whether or not:
- Activities were implemented according to the original plans or undertook new activities;
- Activities were implemented partially or fully;
- There were demonstrated changes in outcomes, including knowledge, attitudes and practices; and
- Project interventions were sustainable.
To comprehensively evaluate the MCA Mongolia Health Project, the Independent Evaluator compared various survey data, reviewed project documents, and conducted interviews with project stakeholders.
|Evaluator||Pim de Graaf, Independent Evaluation Consultant|
|Methodology||Pre-post survey data comparison; ex-post interviews|
|Exposure Period||Between 24 and 30 months for each intervention.|
|Immediate and Intermediate Outcomes||
The evaluator reached the following conclusions (more can be found in Chapter 11 of the full report):
|Ultimate Impact||Assessing the achievement of the ultimate goal of the Health Project, increase of life expectancy of the Mongolian population, is not possible within the time frame of the project or immediately afterwards.|
There were several key lessons learned from this evaluation for MCC and future partner countries to consider when designing and implementing health projects and evaluations:
- Set realistic time horizons. The period of actual implementation of the Health Project was only a few years due to complications in start-up and getting the project into implementation mode, including delays in the Government of Mongolia’s Ministry of Health implementing clinical guidelines necessary for the screening campaign and the major reallocation to the project in year two. This is far too short for a complex program that is so deeply embedded in the health system. Potential delays in implementation should be considered during compact development and an appropriate buffer should be built into the project and evaluation timeline. Due to the short Compact time frame, the training may have been incomplete or insufficient to ensure sustainability of the screening programs. This may have been mitigated with more preparation during the compact development phase. Regardless, with the unexpectedly large reallocation from the Rail Project in 2009, MCC may learn to build stronger plans for course corrections or scale up into compact development.
- Assess and ensure financial sustainability during development phases. In terms of sustainability, the Health Project missed an opportunity to work with the Government of Mongolia on a sustainability plan to ensure appropriate operations and maintenance budget was developed for equipment, and appropriate staffing and training plans were developed for continuation post-Compact. In the future, this should be built into compact implementation and programming.
- Consider social and gender inequalities early on during project design. In this project, certain issues were not identified and addressed until midway through project implementation. These included the higher traffic accident risks for males, which ultimately led the project to implement some education campaigns specifically targeting males. Mobility issues that might prevent disabled people from benefitting were also only properly addressed in the final year of the compact through a targeted campaign for disabled populations. A comprehensive examination of such risks and constraints earlier on would have enabled a more thorough response.
- Assess coordination challenges with partners; set benchmarks and milestones. There were many unexpected challenges the project encountered, including upgrading the State Hospital N.3’s Stroke and Cardiac units in order to equip them for high-tech diagnostic equipment, the installation of three gas management lines in the units for sustainability purposes, and general coordination of all parties involved in the project activity. In many cases, these activities required coordination across a range of stakeholders, including MCC, MCA, Government of Mongolia, and other donors. In the future, MCC will work to assess what activities require coordination across stakeholders, define a coordination strategy, with benchmarks and milestones that are tracked between stakeholders in order to mitigate completion and results risks dependent on coordination across many stakeholders.
- Structure evaluation to ask about sustainability and behavior change. There was a missed opportunity in assessing how well the behavior change component of the project worked and which aspects of the public awareness campaign had the most impact. The scope of the evaluation did not cover this, primarily because the evaluation was completed directly after the project ended rather than after one or more years to observe behavior changes. It would have been helpful for the health community to understand what outreach activities worked well and which did not have as much success. Also, a later evaluation would have permitted a more substantive assessment of the financial sustainability of the project investments which would have been useful for ERR and beneficiary impact analysis.
The evaluation is complete and there are no planned next steps.
- 1. These tables reflect data from the final approved closeout MCA-Mongolia Indicator Tracking Table (ITT). Please note, the Evaluation Report references data from earlier ITTs which may explain some discrepancy.