Measuring Interim Results of the Indonesia Participatory Land Use Planning Activity

In Context

The Millennium Challenge Corporation (MCC) compact with Indonesia is a five-year investment (2013- 2018) of $600 million in 3 projects: Community-Based Health and Nutrition to Reduce Stunting, Procurement Modernization, and Green Prosperity (GP). The Green Prosperity Project has a budget of

$332 million and includes four major activities: Participatory Land Use Planning (PLUP), Technical Assistance and Oversight, Green Prosperity Facility, and Green Knowledge. The $43.1 million Participatory Land Use Planning Activity is the subject of an independent performance evaluation; the interim evaluation report was released by MCC in July 2017 and the results are summarized here. This component represents 7 percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.

Pie Chart: Green Knowledge Spending

*These figures are based on MCC obligations as of December 2016.

Program Logic

The PLUP Activity was originally designed to ensure that projects funded by the GP grant facility were designed on the basis of accurate and appropriate spatial and land use data and adhered to and reinforced existing national laws, regulations, and plans. It was also intended to strengthen the capacity of local communities and district level institutions to manage their own land and resources and encourage investment. The PLUP Activity consists of the four core tasks:

  • Task 1: Participatory (with the community and local government) determination, geo-location, and physical demarcation of village boundaries (village boundary setting, or VBS), the mapping of natural and cultural resource areas within the villages (resource mapping, or RM), and the creation of geo-spatial databases of the information collected—VBS/RM.
  • Task 2: Acquisition of geo-spatial data and preparation of Geographic Information System (GIS) and information management system (IMS) databases for land use/land
  • Task 3: Compilation and geo-referencing of existing and pending licenses and permits for land and natural resource
  • Task 4: Enhancement of district spatial plans through capacity building and spatial planning, enforcement and management of land-use information in spatially-enabled databases.

However, midway through the compact, the Activity implementation plan was altered in a way that affected the logic; PLUP implementation no longer preceded the identification of Green Prosperity investments and it became more of a safeguards program in areas where investments were already planned to occur, rather than a foundational measure designed to attract Green Prosperity investments. The program logic had to be reworked to define the benefits of the PLUP interventions independent of the GP investments.  As depicted in the program logic diagram below, the expectation for PLUP is that demonstrating a participatory process for achieving spatial certainty (i.e. clarity about boundaries and the location and allowable or assigned use of community and natural resources) at the village-level will contribute to improved spatial certainty across the district. Additionally, improved clarity and transparency in documenting geospatial data and land permits and licenses at the district level will also improve spatial certainty and efficiency in permitting and licensing.  Improved spatial certainty overall is expected to result in more and more sustainable private investment, which should in turn result in higher incomes, growth, and poverty reduction. PLUP is expected to be implemented in up to 45 districts through 7 or 8 different implementation contracts, and implementation is still underway.

Program Logic Flow Chart

For a more detailed discussion of the program logic, please refer to the Indonesia Compact M&E Plan, which can be found here: https://www.mcc.gov/where-we-work/m-and-e/indonesia-compact.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program at the time of evaluation data collection.

 

Indicators

Level Baseline (2013) Actual Achieved (12/2016)  

Target

Percent Complete
Number of village boundaries established Outcome 0 0 450 0%
Number of district-level databases containing comprehensive information on land use, land cover, permits and licenses  

Outcome

 

0

 

0

 

45

 

0%

Number of land issues identified Output 0 9 No Target N/A
Land area of villages delineated via VBS (Hectares) Output 0 582,901 No Target 1 N/A
Number of villages assisted in participatory village boundary setting and resource mapping  

Output

 

0

 

114

 

450

 

25%

Number of district-level inventories of land use, land cover, and permits and licenses inventories created  

Output

 

0

 

13

 

45

 

29%

Number of enhanced district-level spatial plans Output 0 13 45 29%
Stakeholders trained Output 0 1,127 2,700 42%
Number of districts that formally adopted guidelines for participatory village boundary setting  

Process

 

0

 

4

 

45

 

9%

Source: Q15 ITT from December 2016, which includes data as of December 10, 2016, based on reporting from PLUP implementers.

 

Evaluation Questions

The evaluation was designed to answer the following questions:

  1. How has PLUP progressed in the achievement of short-term outcomes (1-6), and how is it likely to progress in the achievement of long-term outcomes (7-9)?
    • Outcome 1: Increased public perception of spatial certainty associated with boundaries and land uses within the PLUP villages
    • Outcome 2: Decreased conflict between villages (or groups of villagers from adjacent villages) over land use rights in “border”/outlying areas between villages
    • Outcome 3: Improved confidence in land governance administration within PLUP stakeholder partner institutions
    • Outcome 4: Increased capacity of PLUP institutional stakeholders to manage land and natural resources
    • Outcome 5: Improved land use planning within PLUP locations
    • Outcome 6: Increased conformance of land use (particularly as measured by new project or uses) to the (new/improved) land use plans
    • Outcome 7: Accurate and locally accepted spatial and land use data
    • Outcome 8: Shared understanding of boundaries and various land uses among PLUP geo-spatial partners and communities
    • Outcome 9: Greater efficiency in land permitting/licensing processes
  2. Were achievements toward identified PLUP outcomes varied by geography, community type, or gender and vulnerable/marginalized groups?
  3. What were the main challenges in managing PLUP?
  4. What were unintended results (positive or negative) achieved by PLUP?
  5. Through what pathways, if any, is increased spatial certainty likely to increase household incomes? What evidence does the evaluation find for this?

 

MCC did not conduct a cost-benefit analysis of the PLUP Activity, therefore the evaluation design does not link to an economic model. More detail on this topic can be found in the Evaluation Design Report here: https://data.mcc.gov/evaluations/index.php/catalog/180.

 

Evaluation Results

The evaluation was designed to measure PLUP results in a sub-sample of the full program area, which comprises the first 4 districts where PLUP was being implemented. In order to inform analysis of potential geographic differences in implementation or perceived outcomes, data was collected in all 4 districts, 6 of the sub-districts, and 11 villages across the sampled areas.

Table: Sample Areas

The first round of data collection was planned to occur shortly after the completion of implementation, however due to project delays, data collection took place as implementation was nearing completion. Most outcomes were not expected to have been impacted immediately by the program, therefore the post-implementation data provides a valid baseline. For all other outputs or outcomes, the evaluation team asked retrospective questions about the pre-program period to establish baselines. The evaluation employed three types of primary data collection: Key Informant Interviews (KIIs), Focus Group Discussions (FGDs), and Participant Observations. Review of project documents, legal and planning documents, and news media articles also informed analysis. In total, the evaluation team spoke with 232 respondents through 66 KIIs, 22 FGDs, and 9 observations.

 

The primary purposes of the first round of the PLUP performance evaluation were to establish a baseline for the expected PLUP outcomes and to investigate the link between PLUP and the rest of the GP  Project. A secondary purpose was to assess the performance of the first round of PLUP implementation. In these ways, this report documents both baseline conditions and interim evaluation findings. This summary reports only the interim evaluation findings.

 

Evaluator Social Impact
Impact or Performance? Performance
Methodology Qualitative Pre-Post
Evaluation Period ·         PLUP implementation period (in evaluation areas only): July 2015- March 2017

·         Data collection period: September 2016

·         Exposure period: Data was collected shortly before the completion of implementation in evaluation areas

Outcomes Assessment of implementation

Evaluation Question 3:

·         The most commonly mentioned barriers to the management of PLUP had to do with resources and unclear expectations between those responsible for managing parts of the project – MCC, Millennium Challenge Account – Indonesia (MCA-I), and the implementer.

·         The implementer was asked to deliver unanticipated requirements mid-project, including: Installation of pillars (Task 1); Final stakeholder signatures on maps (Task 1); IMS data sharing functionality (Task 4); and IMS public portal functionality (Task 4).

·         MCA-I and MCC respondents cited management challenges around the delivery of the Operations Manual and implementation of Phase I of the VBS/RM, both critical elements of the contract where the Contractor initially under-delivered.

·         12 months did not allow for activities beyond those that directly related to the delivery of outputs. The implementer had no time (nor scope in the contract) for capacity building regarding utilization of outputs, for example.

 

Assessment of outcomes

Evaluation Question 1: Overall:

  • The VBS/RM process was found to be relevant for stakeholders from the village to the national level, as they commonly noted experiencing challenges related to boundaries and use of land and
  • Stakeholders ranging from the regional planning and development body (BAPPEDA) to investors reported that Task 2 – 4 outputs (or tools) were relevant and important to them.
  • Many project outputs had yet to be delivered at the time of the evaluation, and respondents were unable to fully articulate how these outputs (including maps) would be used in the

By Outcome:

  • Outcome 1: Evidence of increased spatial certainty was found, as measured through awareness and understanding of village
  • Outcome 2: Evidence of a decrease in conflict was found, although complex, ongoing conflicts continue to plague intervention villages. An increase was detected in village-level dispute resolution resources, dispute resolution mechanisms, and agreed boundary segments, all of which were facilitated by the PLUP activities.
  • Outcome 3: Evidence of improvements in land governance administration was found, most prominently at the village level, which likely reflects the completion of PLUP outputs However, the unrealized delivery of critical outputs at the district level has led to less improvement, relative to the village. At the national level, the findings highlighted a sustainability risk stemming from lack of clarity on roles and responsibilities and coordination mechanisms among government stakeholders that are critical to making use of or sustaining PLUP interventions.
  • Outcome 4: Villagers noted that resource mapping discussions helped them think about broader land management District stakeholders received training on systems (e.g. GIS and IMS) that can improve management of land and resources by improving their understanding of area potential, but they do not yet have access to tools for managing, monitoring, and promoting area resources. Accordingly, no evidence was found of stakeholders utilizing tools to manage, monitor, and promote area resources.
  • Outcome 5: Evidence was found that the project mapping and identification of issues had influenced some aspects of spatial planning with respect to development plans; however, at the time of the evaluation the maps had not been finalized nor returned to the villages due to a bureaucratic process requiring a District Head
  • Outcome 6: The evaluation identified instances where PLUP activities have both increased understanding and knowledge about how to adhere to land use plans (village level) and increased stakeholders’ appetites for improved data and information that can be used to better adhere to land use plans (district level).
  • Outcome 7: Too early to assess
  • Outcome 8: Too early to assess
  • Outcome 9: Too early to assess

 

Evaluation Question 2:

  • Though the project was implemented in significantly different geographic, ecological, cultural, and religious landscapes, outputs do not vary across provinces or
  • Final outputs (and progress against outcomes) were not found to vary across implementation phases or implementing partners; however, each phase was unique in terms of length of time and challenges encountered and each implementing partner used unique approaches to complete PLUP
  • The implementation of PLUP activities was male-dominated in that the delivery of outputs involved more men than It is not possible to determine at this stage whether men and women benefited from the resulting outputs equally or not, considering the status of delivered outputs.

 

Evaluation Question 4:

  • Respondents noted that the VBS/RM process provided them clarity on several issues regarding citizen-government relations, including issues like clarification of Indonesian Identity Cards and about where citizens are to pay taxes.
  • Respondents expressed increased pride in their village as a result of the VBS/RM process. The process reinforced cultural identity in many areas and increased a sense of belonging among villagers. This was a result particularly for youth in the
  • At the time of the evaluation, replication was reported by respondents in Mamasa, Mamuju, and adjacent locations to these two
  • One unintended, negative result is the re-opening of previously unresolved disputes in each of the sampled districts, which still could not be concluded with the assistance of PLUP contractors.

 

Evaluation Question 5:

  • The evaluation identified three specific pathways through which spatial certainty is likely to increase household income: 1) at the village level, spatial certainty provides pathways for supporting stronger forms of tenure; 2) village-level spatial certainty also facilitates access to government programs and funding, which can support improved development outcomes for the household; and 3) spatial certainty can also support the larger investment portfolios among central, provincial, and district
  • Primary concerns noted by investors were risk and ways to minimize risk to maximize return on Respondents explained that they see clarification of boundaries and land use/claims (i.e. spatial certainty) as a way to minimize risk. This not only minimizes risk for new investors but also for existing investors.

Assessment of risks

1. PLUP design and approach

  • There is a risk to the long-term achievement and sustainability of PLUP investments due to the conceptualization of PLUP outputs as the delivery of boundaries and IMS without further support on the use of these Lack of follow-on support may impede the achievement of targeted PLUP outcomes.
  • Regarding conflict resolution, future initiatives like PLUP need to have an appreciation for a) conflict that can arise as a result of the process, b) the costs of the approach, and c) required human resources and capacity. Boundary disputes, whether new or longstanding, present an opportunity for bridge-building and resolution that often require time, alternative dispute resolution mechanisms, and capacity appropriate to the
  • The costs of a ‘participatory’ endeavor like VBS/RM are important to consider as MCA-I advocates this approach to the Expenditures (technology, community meeting expenses, pillars, etc.) can quickly accumulate, raising questions of cost-effectiveness.
  • Advocacy efforts are critical for PLUP both within GP and with the MCA-I, however, has not communicated clearly about what the project has done and what further implementation contracts hope to achieve.

2. Design and Management of PLUP Implementation Contracts

  • Both implementation of VBS/RM and promotion of utilization of outputs for village-level planning and community development requires time, likely beyond the 12-months initially allotted. PLUP contractors need time to build community capacity.
  • The first phase of PLUP implementation highlighted the difficulties in being gender inclusive (including reaching marginalized/vulnerable groups), in part due to the high level of effort and time required to engage villages in the VBS/RM process and community Given the number of tasks that needed to be accomplished in a limited amount of time, implementers found it challenging to organize activities at times and venues that were accessible to women in some communities.
  • Resource mapping is seen to be highly relevant to MCC, MCA-I, and investors and the overall logic of PLUP; however, attention to this component of the VBS/RM process seemed under- emphasized and unclear.

3. Coordination of Closeout and Sustainability

  • Closeout plans were not adequately incorporated into PLUP design and implementation up front.
  • Only a minority of interviewed PLUP stakeholders have planned for utilization of the outputs provided by PLUP (e.g. boundaries, maps, IMS, etc.), while the majority are either waiting for further assistance or still require more training or capacity strengthening.
  • PLUP was not designed or budgeted to provide the desired follow-on training.
  • The limited capacity of the MCA-I PLUP management team to guide and manage implementation at the time of the evaluation presents a risk, particularly as the scope of PLUP implementation increases.

4.       MCA-I Engagement at the National Level

  • The MCA-I PLUP Team’s engagement with national level stakeholders is critical for the operationalization of the PLUP concept in Indonesia, but was found to be only in the early stages at the time of the evaluation. Strategic collaboration and coordination with government entities such as BAPPENAS, MOHA, and BIG are critical.
Objective-level Outcomes N/A.  There is no specific objective for the PLUP Activity noted in the Compact because it is part of the larger Green Prosperity Project. The Project objectives are: (i) increase productivity and reduce reliance on fossil fuels by expanding renewable energy; and (ii) increase productivity and reduce land- based greenhouse gas emissions by improving land use practices and management of natural resources. These objectives will be assessed in the broader Green Prosperity evaluation work.
Effect on household income attributable to MCC  

N/A.  The evaluation did not assess this outcome because it was not feasible, given the evaluation methodology.

 

Lessons Learned

  • A program design or implementation change midway can create significant challenges both for ensuring and evaluating PLUP was originally designed to lay a foundation of geospatial information to attract investment to targeted geographic areas and morphed into an approach to ensure that already-proposed investments were supported by geospatial information and follow safeguards. The economic argument for PLUP changed with this design change and the program logic had to be redefined. The implications of this fundamental change were not well understood across all team members. This posed challenges for ensuring a match between PLUP implementation (which had already been structured) and the achievement of newly defined results. Similarly, it created challenges for designing a relevant evaluation. More detailed thinking about the theory of change for standalone PLUP benefits may have prevented the risks to use of outputs and sustainability of results highlighted by the evaluation. Therefore, adequate consideration should be given to programmatic changes that affect the theory of change.
  • The significant investment by the Compact to ensure the achievement of PLUP targets may result in interventions that are too expensive for the government to take to MCA-I and MCC are considering ways to improve the efficiency of PLUP activities, based on

 

the differing implementation models employed by the various PLUP implementers.  The experience so far indicates that targeting local, rather than international, firms to manage PLUP implementation would have been a more cost-effective option.

  • More engagement with and dissemination of information to the districts is critical to making them leaders in geospatial data and planning and ensuring the sustainability of PLUP The project team had originally planned to pilot this kind of approach and then scale up across districts, but delays and challenges in contracting PLUP implementation, particularly given the complex and widespread geographic scope of the program, prevented it from doing so.

 

Next Steps

A second and final round of data collection is planned for this evaluation in 2018, which will seek to further verify short-term outcomes and assess the achievement of long-term outcomes. A final evaluation report is expected to be released in 2019.

 

Footnotes
  • 1. There was no target set for this indicator because the exact villages where the village boundary setting will occur was not known at the outset and so the area could not be estimated.