Measuring Interim Results of the Mongolia Peri-Urban Land Leasing Activity - Phase 2

The MCC Compact with Mongolia was a five-year investment (2008-2013) of $285 million in five projects: (i) the Health Project, (ii) the Property Rights Project, and (iii) the Vocational Education Project, (iv) the North-South Road Project, and (v) the Energy and Environment The Property Rights Project included three major activities, Improvement of Land Privatization and Registration System, Privatization of Ger Area Land Plots and Peri-Urban Land Leasing—commonly known as the Peri Urban Rangeland Project (PURP). The $10.1 million PURP activity, is the subject of this independent impact evaluation and represents 3.5 percent of the total Compact. The results described here are interim findings of PURP Phase 2. Other components of the Compact are the subject of forthcoming independent evaluation.

PURP, in coordination with local officials, provided exclusive pastureland use rights to herder groups and promoted improved animal husbandry practices, including sustainable pastureland management and adoption of “intensive” dairy farm practices among the project participants. Wells, fencing and shelter materials, and seeds for fodder crops were also provided to herder groups who wanted them. The shift in practices anticipated to result from this project is expected to increase herd productivity, decrease land degradation and ultimately raise herder income.

The project was implemented in two phases; the first began awarding pastureland leases in September 2010 in areas around Mongolia’s three largest cities (Ulaanbaatar, Erdenet, and Darkhan), and the second began one year later and concentrated on areas surrounding two smaller regional cities (Choibalsan and Kharkhorin). The results described here are interim findings of Phase 2 of the PURP and focus mainly on short-term changes in behavior such as herd management and rangeland use, but also provide a preliminary look at longer-term impacts on outcomes such as household income. Project impacts are expected to manifest over a period of several years and will be assessed in future survey rounds, the final of which is planned for 2017.

Program Logic

PURP was designed to address the problems associated with overuse of rangelands that were being exacerbated by an increase in herd sizes and migration closer to urban areas in Mongolia. The PURP project logic can be split into three major pathways leading from project activities to desired outcomes, each with associated short- and long-term outcomes. The short-term outcomes are generally behavioral changes that the project hoped to bring about, such as reducing overall herd size, and increased usage  of hay for feeding animals. These short-term behavioral changes are then expected to give rise to longer-term outcomes that reflect an improvement in household welfare and environmental sustainability, such as increased income from livestock, and improved pasture quality. Specifics follow, below:

Expected short-term outcomes

  • Increased perceptions of tenure security
  • Improved grazing practices to maintain carrying capacity of land
  • Improved herd composition, particularly an increase in crossbred cows and other more productive cow breeds
  • Increase in production, storage, and use of hay and other prepared fodder Expected long-term outcomes
  • Higher livestock productivity
  • Decreased herd mortality
  • Increase income from livestock
  • Improved pasture quality due to reduction in overgrazing

 

Three major pathways

  1. Improved rangeland management resulting in environmental sustainability
  2. Improved animal husbandry resulting in increased income from animal products
  3. Increased land tenure security resulting in investment in improvements on the land

There were several key assumptions underlying the PURP project logic during the design of the investment:

  • Prevalence of tenure insecurity due to traditional common grazing pastureland rights was causing overgrazing, conflict over land resources and lack of long-term investments;
  • Rangelands in Mongolia, particularly in the peri-urban areas, were overgrazed;
  • Resulting land degradation was a constraint to sustainability of herder incomes;
  • Per land parcel area calculations and lease terms assumed semi-intensive herders would use land within lease for two seasons (spring/winter) while using communal grazing areas for summer/autumn;
  • Provision of wells and long-term leases would allow herders involved in semi-intensive practices to reduce the number of migrations;
  • Government would pass supporting legislation to allow for long-term leases over grazing areas;
  • Switching from traditional Mongolian cattle to improved-breed milking cattle would improve milk yields due to the higher productivity of improved-breed milking cows.

 

For a more detailed version of the program logic, please refer to the Mongolia M&E Plan, which can be found here: www.mcc.gov.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output indicators specific to the evaluated program.

 

 

Indicators

Level Baseline (2008) Actual Achieved (2013)  

Target

Percent Complete
Leases awarded Output 0 387 1 465 83%
 

Household land rights formalized

 

Output

 

0

 

1,315 2

 

1,758

 

75%

Wells completed Output 0 346 420 82%
Stakeholders trained Output 0 2,334 1,515 154%
Number of legal and regulatory reforms adopted Output 0 0 1 0% 3
Conflicts successfully mediated Output 0 9 No Target N/A
Repayment rate by leaseholders Output 0 18.4 % 80 % 23%

Source: Closeout ITT from December 2015, which includes data through the end of the compact, based on reporting from both Phase 1 and Phase 2 of the Peri-Urban Rangeland Project

 

Of the seven output indicators, six had established targets, one of which was surpassed. The average completion rate of for these output indicators was 69.5%. 4

Evaluation Questions

The Peri-Urban Rangeland Leasing Survey (PURLS) was designed to collect basic socio-economic figures as well as information on key herding related outcomes from the households participating in the study. The evaluation was designed to answer the following specific research questions:

  1. What is the causal impact of participation in the PURP on herder incomes, rangeland carrying capacity, and productivity?
  2. What individual and household level characteristics predict higher incomes, rangeland carrying capacity, and productivity due to participation in the PURP?
  3. What individual and household level characteristics predict changes in rangeland and herd management behavior due to participation in the PURP?

The evaluation answered these questions through a randomized controlled trial (RCT). The evaluation randomly assigned, via a lottery, eligible herder groups to either participate in the project (the “treatment” group) or not (the “control” group). Overall 1,042 households were interviewed for the interim survey.  Specifically the sample of households for the interim survey was defined as follows:

  1. Project Households: All households that were part of the 165 randomly selected herder
  2. Comparison Households: All households who submitted an application as part of a herder group but whose group application was not selected via a
  3. Neighbor Households: A randomly chosen subset of neighboring herder households residing on rangeland plots adjacent (within 2 km) to the 329 herder groups.

Evaluation Results 5

The evaluation focused on the first and key research question around causal impacts of participation in PURP.  Although it was too early to see results in long-term outcomes, the evaluation observed significant evidence that expected short-term changes in herder behavior are occurring in the two project areas albeit differently in each area 6 .   This could have been due to differences in environment or implementers in the two areas. Details of these results are summarized, below, and outlined in the chart that follows.

Expected Short-term Outcomes

  • Perception of Tenure Security: the evaluation found no evidence that project households perceived higher land tenure security than comparison households, although Kharkhorin project households were more likely to believe that they could prevent other herders from overgrazing on their
  • Grazing Practices/Maintain Carrying Capacity: Project households in Choibalsan were more likely to reserve a part of their pastureland in case of bad weather but did not adopt other improved grazing practices to improve carrying capacity of This is likely due to Choibalsan not being as overgrazed and degraded as envisioned during project design.

Kharkorin project herders maintained lower yearly pasture load per hectare and attempted to control the overall size of their herds relative to comparison households. There was no evidence in either of the two project areas of adoption of other improved grazing practices, such as changes in seasonal herder migration, within season livestock rotation/relocation.

  • Herd Composition/Cattle breeds: Project households in Choibalsan increased the percentage of improved breed milking cows
  • Fodder production/storage/use: Project households in Choibalsan increased purchase or receipt of hay/fodder compared to comparison households. However neither project area showed increases in use of hay/fodder.

Long-term Outcomes 7

  • Higher livestock productivity: Contrary to expected increases in livestock productivity, especially milk yields, yearly milk yield per milking cow was higher for comparison households than project households in Choibalsan. No effect observed in Kharkhorin.
  • Decreased herd mortality: Animal mortality rates were lower for sheep and goats for project households in Kharkhorin. No effect observed in Choibalsan.
  • Increase income from livestock: No effect observed
  • Improved pasture quality due to reduction in overgrazing: No effect observed

 

It was too early for the evaluation to answer research question 2 and determine the key characteristics that predict long term outcomes due to participation in PURP as long-term results were not yet expected. The evaluation will address this research question in the final results report.

 

The evaluation was able to partially answer research question 1 by offering one key likely factor in predicting changes in the short-term rangeland management behavior and perception of tenure due to participation in PURP. Specifically, the evaluation looked at data correlations and suggested that whether herders in fact were on land that was overgrazed or over the appropriate pasture load, was likely a key determinant of herder behavior change. The project assumption that Choibalsan was overgrazed was in fact incorrect. As such, changes in rangeland management behaviors were more prominent in Kharkhorin, which was determined by project implementers to be overgrazed.  Similarly, herders in Kharkhorin although no significant changes in perceived tenure believed they could prevent herder overgrazing on their parcels. More analysis will be carried out in the final results report to further understand the driving factors in behavioral change.

 

Evaluator Innovations for Poverty Action (IPA)
Evaluation Type Impact
Methodology Randomized Controlled Trial (RCT)
 

 

 

 

 

 

Evaluation Period

Project implementation:

Leases signed-October 2011,

Herder training- November 2011-June 2013 Supplying alfalfa seeds: February 2012-June 2012 Well installation: March 2012-June 2013

Supplying materials for fence and shelter construction: June 2012-December 2012

 

Evaluation:

Baseline-January 2012-April 2012 Interim- May 2014-July 2014

End line- February 2017-April 2017

 

 

 

 

 

 

 

 

 

 

 

Short-term Outcomes

Phase 2 Project Areas Overall:

·         No overall significant effects observed in adoption of better rangeland management practices 8 , improvement of animal husbandry practices, or land access and security

·         However, the majority of households (69.5%) feel the leases are beneficial and there was a decrease from 20% to 15% of project households reporting livestock numbers that exceeded the carrying capacity estimated by PURP.  Although there is no comparable information for comparison herder groups, we could take this change in pasture load as suggestive evidence of movement toward a sustainable number of animals.

Kharkhorin:

·         Adoption of better peri-urban rangeland management practices-Project households maintained lower yearly pasture load per hectare and attempted to control the overall size of their herds relative to comparison households

·         Improvement of animal husbandry practices- Project households significantly increased the percentage of improved breed milking cows relative to comparison households

·         Increased land access and security-Project households were more likely to believe that they could prevent other herders from overgrazing on their pastureland

 

Choibalsan:

·         Adoption of better peri-urban rangeland management practices: Project households were more likely to reserve a part of their pastureland in case of bad weather, although there was no similar pasture reserve pattern observed for the purpose of rehabilitating the land.

·         Improvement of animal husbandry practices: No observable shift of project households using hay or fodder but project households increased by 14% the likelihood of households purchasing or receiving hay or fodder as a gift.

·         Land access and security: no effects observed

Long-term Outcomes All Phase 2 Project Areas:

*No overall effects seen on livestock productivity or land degradation; however these are longer-term effects which aren’t expected until end line survey.

 

Kharkhorin:

·         Higher Livestock Productivity: Lower animal mortality rates for sheep and goats

 

Choibalsan

·         Lower Livestock Productivity: Contrary to project logic expectation annual milk yields actually were lower per milking cow in project households compared to comparison households.

Objective-level Outcomes The activities are meant to trigger a shift in rangeland management practices that are anticipated to increase herd productivity, decrease land degradation and ultimately raise herder incomes.   Land degradation is being measured by a separate land degradation survey, but no effects have yet been seen. Final results are expected in 2018.
Effect on household income attributable to MCC In the Choibalsan area, there are no detectable differences in the changes of any of the components of earned income between project and comparison households. Project households increased both net livestock income and net earned income by more than comparison households, but these effects were not statistically significant at conventional levels.

 

 

 

In Kharkhorin, project households had a significantly smaller increase in labor cost than comparison households. Project households also had a relatively smaller (but marginally significant) increase in non-livestock earned income.

 

Lessons Learned

Project

  • A better understanding of land quality should be gathered prior to If the land is not overgrazed or degraded, then although improved herd management and marketing practices may be adopted, practices on land grazing and lowering herds to the carrying capacity are not as likely to be useful or adopted.
  • Different assumptions and strategies are required for peri-urban areas that are around large urban centers versus smaller regional town The more rural areas are unlikely to manifest program effects as rapidly as those right outside urban areas, since those rural areas follow traditional semi- intensive herder practices and hence less cattle and related dairy production, as well as being more removed from markets.
  • Payments/Repayment rates are low and difficult to obtain and likely require further investment on land institutions and sustainability.
  • Lease terms and herder business plans should be documented and monitored through local government officials or risk lack of clarity on parcel and herder Again this requires further building during compact activities of institutions and not solely keeping this documentation at the project implementer and contractor level.

Evaluation

  • The length of time between baseline and follow-up should be based on the project logic and expected timeline of results rather than timing of Compact The original timeline for the evaluation was extended from end of Compact in 2013 to end line in 2018 to allow for short- term and longer-term effects to manifest themselves. IPA recommends waiting at least three to five years after the end of the compact for an additional round of data collection, which should allow for the best understanding of project impacts, as the effects are likely to continue to grow as time goes on, and the survey attrition rates have been very low.
  • When designing survey instruments, evaluators should ensure can collect key factors, including land parcel size, grazing patterns and land quality. Through IPA’s collaboration with the USDA/ARS, land quality and productivity was incorporated into the evaluation via USDA/ARS land quality Measuring grazing patterns using animal tracking devices and/or aerial drones in the future could contribute to data collection efforts.
  • RCTs can increase transparency and are feasible in the land sector. An RCT design although not well received in the beginning was later lauded by local stakeholders as an effective mechanism for providing land rights, including being seen as a transparent process by the In cases where the project has limited funding and/or land available for project treatment, randomization via a lottery can work even in the land sector.
  • Project implementers need to keep and provide accurate records of land implementation (timing and outputs), as well as geospatial location of project areas and parcels. The project implementer should be required to regularly provide detailed project data, including the timing, outputs and specific location and names of beneficiaries of its activities, in a mutually agreed upon format with established standards. Moving forward, documentation created by projects should be scanned, digitized and managed.

Next Steps

A final round of data collection is planned for this evaluation in 2017, which will seek to assess the longer-term project effects on herder behavior, livestock productivity and related changes in income.  A final evaluation report is expected to be released to the public in 2018.

Footnotes
  • 1. 155 of leases awarded were from Phase 2. Originally 165 herder groups selected but 10 dropped out.
  • 2. 541 of households were from Phase 2.
  • 3. The Project in coordination with local stakeholders drafted proposed land legislative amendments; however, Parliament did not pass this legislation by the end of the Compact.
  • 4. These figures are calculated using the key non-evaluation indicators with targets in the Property Rights Project/Peri Urban Land Leasing Activity.
  • 5. The survey from which the results described here were taken was conducted approximately 2.5 years after the provision of leases in the Phase 2 areas, so it corresponds well to the timeframe of the short-term outcomes. Long- term outcomes are expected to manifest on a longer time horizon, at least three to five years after the start of project activities. The remainder of the household analysis will be organized around these short- and long-term outcomes.
  • 6. There are no project level significant results. Results were only significant when analyzed at the level of individual project areas.
  • 7. Interim evaluation data was collected during Project implementation and long-term effects were expected a few years post Compact.
  • 8. Positive project impact on investment in infrastructure was observed, but the result was limited to animal shelters, which were provided by the project and hence should be considered outputs.