Measuring Interim Results of the Mongolia Peri-Urban Rangeland Project - Phase 1

The $10.1 million Peri-Urban Land Leasing Activity, commonly known as the Peri-Urban Rangeland Project (PURP), in Mongolia accounted for 3.5% of the total compact. The PURP aimed to increase herd productivity, decrease land degradation and ultimately raise herder income by providing exclusive pastureland use rights to herder groups and promoting improved animal husbandry practices. The results described here are interim findings of Phase 1 of the PURP and show some evidence that herder behavior is changing, particularly observed through a shift in herd composition toward improved breed milking cows, reduced herd size, reduced mortality of sheep and goats, increased likelihood to grow fodder crops, and increased investment in immovable property. The shift toward improved breed cattle and more use of fodder is a crucial short-term behavioral impact that is expected to produce large returns in the form of higher income in the future, although at this stage no significant project impacts were found on livestock productivity or household incomes. The only notable negative interim result is a possible increase in conflicts, which is isolated to just one project area. Due to the nature of the project, it will take time for the full effects of the treatment to be realized, thus longer-term impacts will be addressed more thoroughly in the endline data collection planned for 2017.

In Context  1

Pie Chart: Distribution of Mongolia I Compact Funding

The Millennium Challenge Corporation (MCC) signed a five-year compact with Mongolia, which was a five-year investment (2008-2013) of $285 million in five projects: (i) the Health Project, (ii) the Property Rights Project, and (iii) the Vocational Education Project, (iv) the North-South Road Project, and (v) the Energy and Environment Project. The Property   Rights   Project,   included   the $10.1 million PURP investment, which accounted for 3.5% of the total compact.

PURP, in coordination with soum and bagh officials, provided exclusive pastureland use rights to herder groups and promoted improved animal husbandry practices, including sustainable pastureland management and adoption of “intensive” dairy  farm  practices  among  the  project participants. The project was implemented in two phases; the first began awarding pastureland leases in September 2010 in areas around Mongolia’s three largest cities (Ulaanbaatar, Erdenet, and Darkhan), and the second began one year later and concentrated on areas surrounding two smaller regional cities (Choibalsan and Kharkhorin). The results described here are interim findings of Phase 1 of the PURP and focus mainly on short-term changes in behavior such as herd management and rangeland use, but also provide a preliminary look at longer-term impacts on outcomes such as household income. Project impacts are expected to manifest over a period of several years and will be assessed in future survey rounds, the final of which is planned for 2017.

Program Logic 2

The PURP project logic can be split into three major pathways leading from project activities to desired outcomes, each with associated short- and long-term outcomes. The short-term outcomes are generally behavioral changes that the project hoped to bring about, such as reducing overall herd size, and increased usage of hay for feeding animals. These short-term behavioral changes are then expected to give rise to longer-term outcomes that reflect an improvement in household welfare and environmental sustainability, such as increased income from livestock, and improved pasture quality. The three major pathways are:

  1. Improved rangeland management resulting in environmental sustainability
  2. Improved animal husbandry resulting in increased income from animal products
  3. Increased land tenure security resulting in investment in improvements on the land.

 

Program Goal

Poverty Reduction Through Economic Growth

Post Compact/ Longer-term Outcomes (3-5 years)

Increased herder group incomes from livestock productivity

  • Milk yields and related sales
  • Meat and other non-dairy animal products
  • Decreased mortality

Avoidance of cost of land degradation and cost of feed

  • Increase in plant basal cover
  • Decrease in bare ground
  • Increase in residual biomass
  • Improved forage production per Ha
 

End of Compact/ Shorter-term Outcomes

(1-2 years)

Awareness & Adoption of better peri-urban land use

& rangeland management

  • Adoption of rotational grazing
  • Maintain carrying capacity of land

Awareness/ Adoption/ Improvement of animal husbandry practices

  • Improved herd quality & composition
  • Utilization of more non- forage animal feed (hay production and storage)
 

 

 

Increased land access & security from lease

 

 

 

Outputs

 

Identified and mapped land parcels

Leases provided to herder groups for semi- intensive and intensive parcels

 

Wells, seed and fencing materials provided to project herder groups

Officials & project selected herder groups trained in all project areas

Public outreach events, newspaper articles published & TV programs

Recommendations on draft pasture law, land law & amendments to laws introduced to Parliament

 

 

Activities (sub-activity for Mongolia)

 

Rangeland mapping to identify suitable land tracts

Introduce system of leasing, including policy change, selection criteria and selection

Training for all project herder groups and state officials on livestock marketing & management, rangeland management, & cooperative activities

Provision of seed, fencing materials

& wells to some project herder groups

 

 

 

Public Outreach

Legal and Regulatory Committee and Discussions with Working Group with government Stakeholders

 

There were several key assumptions underlying the Peri-Urban Land Leasing program logic during the design of the investment:

  • Prevalence of tenure insecurity due to traditional common grazing rather than private land rights, which causing conflict over land resources and lack of long-term investments;
  • Provision of wells and long-term leases would allow herders involved in semi-intensive practices to reduce the number of migrations  and intensive project herders to stop seasonal migration;
  • Per land parcel area calculations and lease terms, semi-intensive herders would use land within lease for 2 seasons (spring/winter) while using communal grazing areas for summer/autumn and intensive herders would use land for 3-4 seasons;
  • Government would pass supporting legislation to allow for long-term leases over grazing areas;
  • Rangelands in Mongolia, particularly in the peri-urban areas, had been subject to overgrazing in recent years and had become degraded as a result; and
  • Switching from traditional Mongolian cattle to foreign-breed milking cattle would improve the productivity of milking cows

Measuring Results

MCC uses both monitoring and evaluation data for measuring and reporting results. Monitoring data is typically generated by the program implementers and specifically covers the program participants who received treatment through the compact. However, monitoring data is limited in that it cannot demonstrate what program participants would have done in the absence of the MCC-funded intervention. Accordingly, MCC invested in an independent impact evaluation to establish a counterfactual in order to assess what would have happened in the absence of the MCC investment.

Monitoring Results

The following table summarizes performance on output indicators specific to the PURP Phase 1 and Phase 2 activity according to the compact monitoring data. 3   Output data is provided by MCA/project implementers.

 

Indicators

 

Level

 

Baseline

Actual Achieved

 

Target

Percent of Target Achieved

Leases awarded

Output

0

387

465

83%

Household land rights formalized

Output

0

1,315

1,758

75%

Wells completed

Output

0

346

420

82%

Stakeholders trained

Output

0

2,334

1,515

154%

Number of legal and regulatory reforms adopted

Output

0

0

1

0%

Conflicts successfully mediated

Output

0

9

No target

N/A

Repayment rate by leaseholders

Output

0

18.4

80

23%

Of the seven output indicators, six had established targets, one of which was surpassed. The average completion rate of for these output indicators was 69.5%.

Evaluation Questions

The research questions addressed by this evaluation are the following:

  1. What is the causal impact of participation in the PURP on herder incomes, rangeland carrying capacity, and productivity?
  2. What individual and household level characteristics predict higher incomes, rangeland carrying capacity, and productivity due to participation in the PURP?
  3. What individual and household level characteristics predict changes in rangeland and herd management behavior due to participation in the PURP?

Interim Evaluation Results

The PURLS Phase I follow-up data shows some evidence that herder behavior is changing, which is a necessary condition for other more fundamental effects to take place, such as increases in income. Positive results relative to comparison households were illustrated through a shift in herd composition toward improved breed milking cows, reduced herd size, reduced mortality of sheep and goats, increased likelihood to grow fodder crops, and increased investment in immovable property. The only notable negative result is a possible (and statistically weak) increase in conflicts, which is isolated to Ulaanbaatar area. Many results varied based on the project area (Ulaanbaatar, Erdenet, Darkhan) and by household type (semi-intensive, intensive).

The most significant and robust project impact observed at this stage was a significant increase in the representation of improved breed cattle, which was complemented by a reduced proportion of goats in the herds of project households in all three peri-urban areas compared to comparison households. With regard to herd composition, the percent of cows that were improved breed increased relative to comparison households in the Erdenet area, and for semi-intensive households in Ulaanbaatar. The portion of sheep units made up of improved breed cows likewise was found to have increased in Erdenet, and for intensive households in both Darkhan and Ulaanbaatar areas. These changes are accompanied by a significant relative fall in the percent of goats in the herd for intensive households in all three areas. In Ulaanbaatar, there was evidence that project households, particularly in intensive households, were significantly reducing the overall size of their herds, while in the Erdenet area there was actually a relative increase in herd size (by 53 sheep units) for semi-intensive households.

Also observed in all three areas was a significant increase in the percentage of project households that grew their own fodder crops relative to comparison households. Households in the Darkhan area, as well as the semi-intensive households in Erdenet, fed their cattle with hay or fodder significantly more days in the year.

No impact was found on migration in the combined analysis; however, in the split analysis we find that intensive households decreased the number of seasonal migrations relative to the comparison group in Erdenet and Ulaanbaatar. Moreover, in Darkhan area, semi-intensive households increased the number of migrations relative to the comparisons. Thus intensive and semi-intensive households appear to have been affected differently by the project in terms of migration patterns.

This shift toward improved breed cattle, along with more use of fodder, is a crucial short-term behavioral impact that, according to the project logic, should lead to large returns in the form of higher income in the future. Although no significant project impacts were found at this stage on livestock productivity as measured by yearly milk yield per milking cow, it may just be that the transition was too recent to begin seeing the higher productivity of fully-producing adult improved-breed cows. Likewise, there was little evidence at the time of the follow-up survey of project impact on household incomes. In particular, in none of the areas was there a significant impact on net earned income or net income from livestock, though in Erdenet area project households increased their revenues from selling animals relative to comparison households. In contrast, the project households did significantly increase the costs of their livestock operations relative to comparison households in both Erdenet and Ulaanbaatar areas.

Although it will take time for the full impacts to be realized, some trends towards achieving longer term outcomes have been observed. In Darkhan and Ulaanbaatar areas, there was an improved perception of the land quality at their winter camp by semi-intensive households. It was also observed that project households in both Darkhan and Erdenet areas increased their level of investment in immovable properties (housing and other structures) relative to the comparison households. There was no evidence of a similar pattern in Ulaanbaatar area, which IPA notes is perhaps due to the still uncertain tenure security which was evidenced by the increased number of pastureland conflicts in that area.

Evaluator

Innovations for Poverty Action (IPA)

Evaluation Type

Impact

Methodology

Matching design

Evaluation Period

December 2012-April 2013

Short-term Outcomes

All Phase 1 Project Areas:

  • Increased representation of improved breed cattle in the herds of project households
  • Reduced proportion of goats in herds of project households, particularly by intensive households
  • Increased percentage of project households that grew their own fodder crops

Ulaanbaatar

  • Decreased herd size among project households, particularly by intensive households
  • Intensive households decreased the number of seasonal migrations

Erdenet

  • Increased herd size among semi-intensive project households
  • Decreased haymaking activities
  • Increased number of days that semi-intensive project households fed their cattle with hay or fodder
  • Intensive households decreased the number of seasonal migrations

Darkhan

  • Increased haymaking activities
  • Increased number of days that project households fed their cattle with hay or fodder
  • Semi-intensive households increased the number of seasonal migrations
Long-term Outcomes

All Phase 1 Project Areas:

  • No significant project impacts on land productivity (measured by yearly milk yield per milking cow)
  • No significant impact on net earned income or net income from livestock

Ulaanbaatar:

  • Improved perception of land quality at project households’ winter camp by semi-intensive households only
  • Increased costs of livestock operations
  • Decreased herd mortality rates

Erdenet

  • Increased revenues from selling animals
  • Increased costs of livestock operations
  • Decreased herd mortality rates
  • Increased investment in immovable properties

Darkhan

  • Improved perception of land quality at project households’ winter camp
  • Increased investment in immovable properties

 

Lessons Learned

  1. The length of time between baseline and follow-up should be based on logic and expected results rather than end of Compact. Original timeline for the evaluation was extended to allow for short-term and longer-term effects. IPA recommends waiting at least three to five years after the end of the compact for an additional round of data collection, which should allow for the best understanding of project imapcts, as the effects are likely to continue to grow as time goes on, and the survey attrition rates have been very low.
  2. Analysis of semi-intensive and intensive herders separately would have benefitted from further assessment at the beginning to ensure could create a similar comparison group for intensive herders.
  3. The project implementer should be required to share detailed project data, including the timing, outputs and specific location and names of beneficiaries of its activities,
  4. Land parcel sizes, grazing patterns and land quality are imperative to effective analysis of project results.
  1. An RCT design was planned and would have mitigated any issue of comparability between treatment and controls; however, there were not enough parcels following qualification of project parcels, including a stringent environmental and social review. Phase 2 methodology of parcel selection was changed based on learnings from Phase 1, which allowed for an RCT design via a public lottery.

Next Steps

The PURP evaluation is ongoing with the endline data collection planned for 2017, at which point it is expected that more long-term outcomes will have materialized. For future waves of data collection, numerous changes and additions to the survey instruments will be required in order to more effectively measure changes in all of the key outcomes associated with the project logic. Many of the necessary changes have already been incorporated into the PURLS Phase II Follow-up Survey, which was completed in 2014. Most prominently, endline data collection includes collection of grazing patterns to improve understanding of herd management and pastoral practices.

Footnotes
  • 1. Rail project was cancelled following compact restructuring
  • 2. The survey from which the results described here were taken was conducted approximately two years after the provision of leases in the Phase I areas, so it corresponds well to the timeframe of the short-term outcomes. Long- term outcomes are expected to manifest on a longer time horizon, at least three to five years after the start of project activities. The remainder of the household analysis will be organized around these short- and long-term outcomes.
  • 3. These tables reflect data from the final approved closeout MCA-Mongolia Indicator Tracking Table (ITT) for both Phase 1 and Phase 2 of the Peri-Urban Rangeland Project. Please note the Evaluation Report references data from earlier ITTs which may explain some minor discrepancies in results statements. Post compact reporting may provide additional updates.