Summary of Findings

Measuring Results of the Philippines Community Driven Development Project, Kalahi-CIDSS

View as PDF

The MCC compact with the Philippines was a five-year investment of $385 million. The $125 million Kalahi-CIDSS (KC) Project is the subject of an impact evaluation and a cost study, written up in two reports: “Kalahi-CIDSS Impact Evaluation Third Round Report (Impact Evaluation)” and “KALAHI-CIDSS Subprojects Cost Study (Cost Study.)”  The KC community-driven development project sought to improve residents’ socioeconomic situations, help them become more engaged in local government, and empower communities.   By compact end, KC had financed more than 4,000 small infrastructure projects (including school classrooms, water systems, farm-to-market roads, etc.) and trained community volunteers in more than 3,000 villages.

 

The Impact Evaluation, a five-year randomized control trial with three rounds of surveying paired with real-world observations and qualitative research, found both marked successes and notable areas where no effects were detected. Projects implemented in KC areas, even those not funded by KC, were more reflective of residents’ stated priorities than projects implemented in non-KC areas. Consistent with this finding, residents’ satisfaction with the program was extremely high. Services like roads, education, and water delivered benefits to residents like reduced travel time and costs, reduced agriculture transport costs, increased enrollment, and reduced time and costs to obtain water.  In contrast and unexpectedly, roads projects reduced agricultural productivity. This finding will be the subject of further research by the evaluator.  Lastly, KC was not as effective at generating broader, post-project social changes related to improved governance or community empowerment.

The Cost Study assessed the cost, quality, time frame and sustainability of the infrastructure built via community-driven development (KC) and centrally planned projects (non-KC) that were similar in type and scale.   In terms of cost, for both KC and non-KC actual unit costs were lower than planned.  For quality, KC water projects were of higher quality than non-KC ones, while KC roads were of similar quality, and KC schools were of lower quality.  Overall, the time analysis shows that more non-KC projects were completed early or on time compared to KC.  In terms of sustainability, in most of the sub projects, except the KC water systems, there is little evidence of funding for future operations and maintenance (O&M).

The lessons learned are: The participatory KC process is better than the status quo at identifying residents’ small infrastructure preferences. In the future, CDD-like processes could be incorporated into programs to select, design and/or site small community infrastructure.

KC does not appear to have changed citizen participation in local governance beyond the project.  If this is to remain a key aspect of the CDD theory of change, local political leaders may need to be targeted for capacity building or other interventions as part of the project.

Results were analyzed for different subgroups, women versus men, Indigenous Peoples versus Non-Indigenous Peoples and poor versus non-poor.  The only cases in which KC affected people differently was for Indigenous Peoples (IPs).  For example, IPs appear to benefit substantially more from improvements in access to education than do non-IPs .

The quality and sustainability of community infrastructure, built via KC or other government ministries, should be prioritized over cost and time.  Local design guidelines should be considered along with international standards to determine the appropriate design.  Coupling quality and sustainability, is important, because O&M expenditures are often not financed.  Therefore, the lower the O&M burden because of better quality infrastructure, the higher the likelihood of sustainability.

It may be worth further research to test what implementation modality results in superior infrastructure quality. One can envision that a non-community-driven-development implementation model with heavy citizen engagement and input at the outset could still generate projects that meet community needs.

In terms of the evaluation, there will be further analysis of agriculture productivity related to roads projects to try to understand the counter-intuitive finding that roads projects reduced agricultural productivity.

In Context

The MCC compact with the Philippines was a five-year investment from 2011-2016 of $385 million in 3 projects:  the Kalahi-CIDSS Project (KC), the Revenue Administration Reform Project (RARP) and the Secondary National Roads Development Project (SNRDP). The Kalahi-CIDSS (KC) Project is a nationwide, government-run, community-driven development (CDD) project in the Philippines.  KC pairs community training with block grants at the barangay or village level, which are meant to enable communities to address their self-identified development needs, largely through financing and building public infrastructure and public services called “subprojects.”  The KC project was financed by both MCC ($125 million) and the World Bank ($59 million.)  The KC Project is the subject of two independent evaluations; the results of these studies are summarized here.  The KC component represents 32 percent of the total compact. Other components of the compact are the subject of independent evaluations.
Pie Chart: Philippines Compact Spending
*These figures are based on MCC obligations as of September 2016  

Program Logic

The Philippines lagged significantly behind other countries in the region with respect to government development expenditures as a percentage of GDP and infrastructure investment and quality. The Asian Development Bank’s 2007 growth diagnostic report found that inadequacies in infrastructure were a critical constraint to growth and that the availability of basic infrastructure (water, sanitation, roads, electricity) was regressive. Provision and use of education and health services were found to vary across regions, particularly as a function of incomes.   Community-driven development projects are a strategy for addressing these constraints and providing community empowerment and poverty reduction. In the past, they have been used to support a wide range of community priority needs including provision of water supply and nutrition programs for women and children; building of school classrooms, day care and health facilities, farm to market roads, foot bridges, and drainage systems; and support for productive enterprises such as pre- and post-harvest facilities as well as community capacity building.   Through KC, communities (“barangays” or villages), together with their village and municipal governments, were trained, to choose, design and implement sub-projects that were intended to address communities’ most pressing needs. This was done through a three-year, three-cycle program, which included “social preparation” training for communities, barangays, and municipalities, and sub-project implementation. To address gender concerns, the KC project included dedicated gender staff positions and gender-focused activities, including the provision of “gender incentive grants” to communities.  The KC project funded by MCC was an expansion of an initial KC project ensure (“KC1”) that was implemented between 2003 and 2010. KC1 was funded by a loan from the World Bank.  
Flow Chart: Program Logic
The KC Project included the following activities.
  1. Capacity building and implementation support activities: Millennium Challenge Account-Philippines (MCA-P) provided the staff salaries, logistical support and training for the frontline workers, known as the area coordinating teams (ACTs) at the Department of Social Welfare and Development (DSWD.) The role of the ACTs was to carry out the “Community Empowerment Activity Cycle” (CEAC), each year gradually “handing off” the CEAC to local government. In each year or cycle, barangays held a series of meetings that were facilitated by members of the ACT in which barangay residents identified and prioritized constraints to economic activities within their communities and then identified and prioritized solutions to these constraints. Finally, the barangay presented its solution or subproject to the “Municipal Inter-Barangay Forum” (MIBF) for possible funding. After each cycle or year, there was a transition and reporting period.
  2. Grants for community projects activity: MCA-P funded DSWD grants to implement community-chosen sub-projects. A Gender Incentive Grant initiative was also funded to foster women’s participation in local development.  In addition, the municipalities and barangays provided cash and/or in-kind contributions (including partially-paid labor and local materials) to the sub-projects equal to at least 30 percent of the total sub-project costs. The amount of funding was allocated to municipalities based on the number of barangay within that municipality.
 
  1. Project management activity: DSWD received salaries, logistics and training for DSWD project management staff at the regional and national level.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results 

The following table summarizes performance on output, outcome and objective indicators specific to the evaluated program.
Indicators Level Baseline (2011) Actual Achieved Target Percent Complete
Activity 1: Capacity Building and Implementation Support
Percentage of MLGUs that provide technical assistance in KC sub-project preparation, implementation, and monitoring, based on MOA Outcome NA 100 80 125%
Percentage of Indigenous persons (IPs) that are present during Barangay Assemblies Outcome NA 7.35 No target No Target
Percentage of municipalities that provide their KC Local Counterpart Contributions (LCC) based on their LCC delivery plan Outcome NA 100 80 125%
Number of barangays that have completed all the trainings during the social preparation stage Output 0 3,760 3000 125%
Activity 2: Grants for Community Projects
Number of Subprojects (SPs) with 100% physical accomplishment Output 0 4,011 3217 125%
Number of barangays that have completed specific training on subproject management and implementation Output 0 3,114 1500 208%
Number of Gender Incentive Grant (GIG)-funded Subprojects (SPs) Output 0 55 No target No Target
Source: (Closeout ITT from 09 2016, which includes data through the end of the compact) Targets were exceeded for all the indicators that had targets.  For the “percentage of MLGUs that provide technical assistance”, this increase is mainly programmatic as it is a project requirement for MLGU to provide this assistance.  For the “number of barangays that have completed all the trainings during social preparation phase,” more barangays were reached by the project than targeted, because the average number of barangays in each municipality was bigger than initial estimates.   The total number of SPs with 100% physical accomplishment exceeded its target early on due to bigger counterpart funding from local government as well as favorable currency exchange rates. Toward the end of the compact, funds from other compact projects were transferred to KC, which increased the compact-end target from 2,740 to 3,217.  The Gender Incentive Grants subprojects were introduced in this iteration of the KC project based on various consultations with women stating that even though women had been numerically relatively well represented in KC meetings, the projects funded usually were not those that were the priority of the women.  The objective level indicators were covered in the evaluations, so are not presented above.
  1. Evaluation Section: For the first study, “Kalahi-CIDSS Impact Evaluation Third Round Report (Impact Evaluation)”
This report summarizes the four-year impact findings of the Kalahi-CIDSS (KC) Project, a nationwide, government-run, community-driven development (CDD) project in the Philippines.  These evaluation findings are based on a randomized control trial in which a sample of 198 municipalities across the Philippines’ three main island groupings were randomly assigned to participate in KC or to remain part of a control group for three years.

Evaluation Questions

The evaluation was designed to answer the following questions in three spheres:
  • Socio-economic
    • To what extent did
      • Subprojects (SPs) improve access to related key services?
      • Roads SPs reduce agriculture, fisheries and livestock transport costs?
      • Roads SPs improve productivity in agriculture, fisheries, and livestock sectors?
      • School SPs increase school enrollment and improve student/ teacher ratios?
      • Water SPs reduce time and costs spent obtaining water?
      • KC raise household consumption and asset holdings?
      • KC raise household labor force participation and earnings?
    • Institutional
      • To what extent did
        • KC increase quantity and quality of participation in local governance around decision-making and implementation related to KC activities?
        • KC increase participation in and knowledge of formal structures beyond KC?
        • KC improve barangay information sharing and inclusiveness beyond KC?
        • KC increase confidence and self-efficacy beyond KC?
        • KC increase knowledge and awareness of local governance?
        • KC improve degree to which barangay projects correspond to ex-ante preferences?
        • KC improve perceptions of local governance?
        • KC raise capacity of barangay government?
      • Community empowerment
        • To what extent did
          • KC increase interactions among peers?
          • KC increase participation in community organizations?
          • KC improve how well communities deal with natural disasters and other hardships?
  In addition, the evaluation analyzed whether impacts were different for the following subgroups: women, Indigenous Peoples (IPs), households classified as poor at baseline, villages with baseline levels of better governance, villages whose baseline outcomes were higher, and poor villages. More detail on this topic can be found in the Evaluation Design Report here: https://data.mcc.gov/evaluations/index.php/catalog/59  

Evaluation Findings

Results are analyzed across three domains: socioeconomic, institutional, and community empowerment. Each domain includes multiple hypotheses. Each hypothesis makes up an index or group of indicators. These indices reduce the number of unique chance outcomes, since if the hundreds of unique outcomes were examined, some would be statistically significant due to chance. Results for each hypothesis are presented in standard deviation units. An overview of results is shown in the table below. The table includes the regression coefficient in standard deviation units and statistical significance.
 
Evaluator Innovations for Poverty Action
Impact or Performance? Impact
Methodology                Randomized Control Trial
Evaluation Period The findings reflect data collected between 2011 and 2015 through a variety of methods, including extensive interviews of nearly 6,000 households, barangay (village) leaders and project staff.  Baseline data collection took place, April-June 2012, interim data collection February-June 2014, and third round data collection July-October 2015. Qualitative research complemented the quantitative analysis at baseline and endline to deepen the findings. The study also conducted structured community activities to observe whether KC practices were carried over into other areas of local governance. The Compact ended in May 2016.
Outcomes See text below for interpretation of findings from table
Results of KC across Three Primary Domains
Hypothesis Indicator and index components Coeff/ sig
H1a SPs improve access to related key services  0.14*
H1b Roads subprojects reduce agriculture, fisheries and livestock transport costs  0.17**
H1c Roads SPs improve productivity in agriculture, fisheries, and livestock sectors -0.20*, -0.04**, 0.01
H1e School subprojects increase school enrollment and improve student/ teacher ratios  0.42***
H1h Water SPs reduce time and costs spent obtaining water  0.15**
LT1 KC raises household consumption and asset holdings -0.03
LT2 KC raises household labor force participation and earnings  0.02
H2 KC increases quantity and quality of participation in local governance around decision-making and implementation related to KC activities -0.03
H3a KC increases participation in and knowledge of formal structures beyond KC   0.04
H3b KC improves barangay information sharing and inclusiveness beyond KC  -0.06
H3c KC increases confidence and self-efficacy beyond KC  -0.07**
H4 KC increases knowledge and awareness of local governance   0.28**
H5 KC improves degree to which barangay projects correspond to ex-ante preferences  0.09***
LT3 KC improves perceptions of local governance  0.02
LT4 KC raises capacity of barangay government  0.23
H6 KC increases interactions among peers -0.03
H7 KC increases participation in community organizations  0.31**
H8 KC improves how well communities deal with natural disasters and other hardships  0.04
* denotes significant at 90% confidence level, ** denotes 95%, *** denotes 99%
Objective-level Outcomes
Effect on household income attributable to MCC The study found no effect on household income or consumption, but it may be too early to measure this.  The study did find statistically significant changes on intermediate outcomes like travel times, and calculated a post-project ERR of 3%, based on these intermediate outcomes.
  In the socioeconomic domain, KC achieved implementation goals and was effective at delivering benefits to residents via subprojects. The socioeconomic domain examines impacts of investments in transport and water infrastructure, reducing the time and cost to travel to key services, transport farm product to the market, and to obtain water for domestic use.  The analysis shows benefits from several types of subprojects: road improvements reduced the time and cost to reach key local services (H1a) and to get farm products to market (H1b), education projects improved enrollment in elementary and secondary schools (H1e), and water projects substantially reduced the time and cost to obtain water (H1h).  At the same time, roads subprojects actually reduced agriculture productivity (H1c_ag), and had no effect on fisheries productivity (H1c_fa) or livestock and poultry productivity (H1c_lp). More generally, there is no evidence that KC affected households’ overall poverty status, as captured by their consumption, assets, or housing quality (LT1), or households labor force participation and earnings (LT2), although such gains may yet occur in the coming years (outside of the four years of project implementation captured by our data). The economic rate of return (ERR) analysis considers the magnitudes of the socioeconomic gains from these subprojects relative to their costs.  The large gains from roads, water, and education subprojects are offset by very large losses in rice production due to the roads projects, limiting the project’s overall cost effectiveness.  The ERR is estimated at 3%, low by most standards.  Excluding the rice losses raises the ERR to 28%.  There will be further analysis of agriculture productivity related to roads projects to try to understand the counter-intuitive finding that roads projects reduced agricultural productivity. In the institutional domain, KC improved the responsiveness of local government to community needs and increased knowledge and awareness of local governance. This domain looks at the quantity and quality of participation in governments and decision-making related to KC and beyond KC, residents’ confidence and self-efficacy, knowledge and awareness of local governance, and the degree to which local development projects correspond to preferences stated at baseline.  As a result of KC, development funds (including those from non-KC sources) were more closely aligned with residents’ preferences in more intensely treated areas (H5). This means that KC was not only effective at delivering services to communities, but that it also was effective at delivering services that communities preferred. KC also improved knowledge and awareness of local governance (H4).  As a result of the KC process, residents in KC communities were more familiar with local officials and local governing bodies. This outcome was significant at interim and at third round and increased between the two rounds. At the same time, this improved knowledge was accompanied by worsening perceptions of confidence and self-efficacy, or individuals’ belief that they have the agency and ability to improve a situation (H3c). KC projects were more responsive to community needs and people knew more about local government, but in the end felt less empowered to make changes. Additionally, KC did not have an effect on the quality and quantity of participation in governance around decision-making and implementation related to KC (H2).  This result was significant and positive at interim, but by the third-round survey, it was no longer significant, possibly because, in 2015, control groups began to implement the successor project to KC, the KC-National CDD Program (KC-NCDDP). At the time of the third-round survey, control municipalities had begun to implement the early stages of the CEAC. Participation in governance related to KC, such as participating in or knowing about barangay assemblies, is an early milestone of CEAC implementation, and thus one potential explanation for the lack of significant difference between treatment and control groups at third round. There is no effect on participation in and knowledge of formal structures beyond KC (H3a, H3b). KC’s participatory processes do not appear to have been carried over to other local governance activities, measured both through survey interviews and through our structured community activity. In the community empowerment domain, KC encouraged communities to engage in development activities. Because the KC process requires a good deal of community participation, reasonable concerns arise about the project’s potential to crowd out other civic activities.  Having been exposed to KC actually led these residents to contribute to other civic activities at greater levels, allaying such concerns about crowding out (and even suggesting some crowding in) (H7). This result endured by the third round – it was significant at interim and third round, albeit there was a slight decline between second and third rounds. However, there is no evidence of changes in interactions among peers, meaning the intensity and frequency of interaction with neighbors generally and specifically about problems in the barangay (H6). This result was significant and positive at the interim survey. It’s possible that this ended up being no longer significant (and slightly negative) at third round for several reasons. First, baseline levels of interaction among community members were very high, so there wasn’t much room for improvement. Improvement may have happened in the initial years of the project, but perhaps it wasn’t possible to expand on this further. By third round, as mentioned above, some of the control group communities had already started the early stages of CEAC, so it’s possible that the control group had already started interacting with more with peers by the third round and this is why there’s no detectable difference between treatment and control groups. There is also no effect on the ways communities deal with hardships or natural disasters (H8). At interim, the evaluation considered the potentially buffering effects of KC of communities dealing with the hardship of Typhoon Yolanda and other natural disasters. Specifically, on helping communities deal with the effects of natural disasters, KC had a significant and large effect at interim. When considering all forms of hardship or natural disaster, by third round there was no buffering effect of KC. We assess the extent of variation in the impact in each domain across all subgroups.  At the household level, recall that this is women, Indigenous Peoples (IPs), and households classified as poor at baseline. At the barangay level, this is barangays with baseline levels of better governance, barangays whose baseline outcomes were higher, and poor barangays.  For most of cases, we observe no differential effects in the subgroups. In some cases, these results are due to limited statistical power given that the set of barangay implementing the specific SP type being considered in the hypothesis was too small to reliably estimate impacts after the hypothesis was subset into subgroups. However, even for two of the most common SP types (transport and education), we observe no differential effects for the five out of our six subgroupings. In fact, there are only three cases in which the treatment effects differ significantly between subgroups, all occurring across the IP and non-IP groupings. Overall, the evaluation finds, consistent with earlier work, that CDD is effective at delivering public goods to community members. It’s clear that KC delivered public goods; and those that met citizen priorities. Projects implemented in KC areas, even those not funded by KC, were more reflective of residents’ stated priorities than projects implemented in non-KC areas. Services like roads, education, and water delivered benefits to residents like improved travel time and cost, improved agriculture transport cost, increased enrollment, and reduced time and cost to obtain water; although roads projects reduced agricultural productivity. Consistent with this finding, residents’ satisfaction with the program was extremely high. However, KC was not as effective at generating broader social changes related to improved governance or community empowerment, or changes that persist or spill over beyond the project. Initial indications of KC implementation, such as participation in local governance, increased knowledge or awareness of local governance, and peer interaction had materialized by the interim data collection, demonstrating early implementation progress. (Although indicators related to participation in local governance and peer interactions were no longer statistically different between treatment and control groups by third round, perhaps because of KC-NCDDP implementation.) Yet the expectation that KC would generate participation in local governance or improve barangay information sharing and inclusiveness beyond KC or affect in the long run how individuals coped with hardship or natural disaster did not materialize. Although the coefficient is small, residents in KC areas felt less confident or self-efficacious compared to communities that had not implemented KC. One exception to this is the finding that KC actually strengthened participation in community organizations. Contrary to an earlier study of KC that found that participation in KC was crowding out participation in other community activities, residents in KC areas actually participated more intensely in community organizations as a result of KC. From a methodological perspective, the analysis shown in this report is the first of its kind to show impacts of funding for certain subproject types on specific related outcomes, in addition to the average effect across all treatment barangay. For example, we show that education subprojects specifically improved enrollment in elementary and secondary schools. This is a methodological advancement in terms of understanding the effects of CDD programs.  
  1. Evaluation Section: For the second study, “KALAHI-CIDSS Subprojects Cost Study (Cost Study.)”

Evaluation Questions

The main objective of this cost study was to assess the cost, quality, construction time and sustainability of the infrastructure built via community-driven development (KALAHI-CIDSS – KC) and centrally planned projects (non-KC) that are similar in type and scale.  The overarching question was: Do community-driven development (CDD) projects deliver similar quality infrastructure as centrally-planned infrastructure faster with lower cost, and superior sustainability?  The evaluation compared similar projects (schools, water systems, roads and environmental protection) implemented either by KC or other government ministries.  Below are the main questions.
  • What is the difference in construction time for KC and non-KC funded community infrastructure?
  • What is the difference in quality for KC and non-KC funded community infrastructure?
  • What is the difference in cost for KC and non-KC funded community infrastructure?
  • What is the difference in sustainability for KC and non-KC funded community infrastructure?
This evaluation was managed by MCA Philippines and not MCC.  MCC contracted an independent peer review of the evaluation.  This peer review along with other evaluation documents and datasets can be found here:  https://data.mcc.gov/evaluations/index.php  

Evaluation Findings

This evaluation is a cost study and compared similar projects (education, water, farm-to-market roads and environmental mitigation) implemented by KC and other government ministries.  The evaluators relied on document review, information from the relevant ministries’ information systems, meetings with stakeholders and site visits to the infrastructure.
 
Evaluator AARC
Impact or Performance? Cost Study
Methodology                Comparison of similar infrastructure built by either DSWD (via CDD methodology) or by other government ministries.
Evaluation Period The study was designed to assess completed infrastructure including sustainability of the infrastructure, therefore, the study began in the last year of the compact.   The methodology for the study was accepted in January 2016.  Field data was collected from February to March of 2016 and the draft report and presentation of findings occurred in April 2016.  The Compact ended in May 2016.
Outcomes Objective-level Outcomes ·         Time to complete a subproject (adjusted for size of subproject) ·         Quality of subprojects: quality ratings were designed by the evaluators, are specific to the types of subprojects (roads, schools, water and environmental protection) and were used to assess each subproject.  The study team developed a list of definable features of work (DFOW’s) for each SP type. DFOW’s are features that are observable, relevant to the quality and safety of the project in question, and common between types of SP’s. While the DFOW list does not constitute the comprehensive checklist of work items that would be produced for a standards-based review, enough features were assessed for each SP to make reasonable comparisons. ·         Cost of subprojects included pre-construction, construction, post-construction (including operations and maintenance) and management costs.  All costs were converted to 2013 prices to control for inflation and were converted to unit costs to control for differences in scale, see table below.  
Infrastructure Type Unit Cost
School Buildings Area of building (m2)
Farm-to-Market Roads Volume of road surface (m3)
Water System Length of installed pipes (m)
Flood Protection Volume of protective barrier (m3)
·         Sustainability refers to the likelihood that a given SP will deliver benefits over its economic life and included the following:
SP Sustainability Facets and Variables
Economic Facet Social Facet Environmental Facet
§  O&M contribution from local government(s) §  Review of O&M plan §  Community organizations’ contribution for SP O&M §  Existence of a system of user-fee collection from SP use §  Contribution from SP user organizations §  Transparency in the use of O&M funds §  Improvement to SP post construction §  Status of active infrastructure maintenance schemes in the community §  Status of community organizations in looking after infrastructure §  Availability of community volunteers for O&M §  Availability of skilled labour locally for O&M work §  Status of O&M team §  Sign of elite capture §  Use of infrastructure for other community purposes §  Emergency response arrangement §  Safety issues associated with SP  
Effect on household income attributable to MCC Not applicable.  The other study, Kalahi-CIDSS Impact Evaluation Third Round Report (Impact Evaluation)” included measures of impact on households and calculated a post-project ERR of 3%.
Communities Are Benefitting from Increased Access to Services
  • It was evident that the communities are benefitting from increased access to services provided by the implemented SPs. Overall, each infrastructure was being used by the community in remote rural communities.
All SPs Encountered Inconsistent Implementation
  • It is clear from the Study that all SPs are challenged by at least one of the four factors, time, quality, cost and sustainability and usually more than one. This reinforces a widely recognized principle that achieving only two is realistic.
  • The resulting impact of the project constraints is that SP implementation is inconsistent across all modalities (modalities include National Government Agency, Bottom-Up Budgeting, Performance Challenge Fund, Local Government Unit and KC). Some SPs may be average quality but with a high likelihood of sustainability or overdue on time but on or under budget.  The extent of these inconsistencies suggests scope for improvements so that SP implementation on a nationwide basis can be strengthened.  Given that the project constraints are encountered by both modalities (KC and nonKC) in this Study, the inconsistencies suggest that prioritizing two factors or project constraints for improvement would yield results.
Overall Time Findings
  • The time to implement a SP is a critical factor as faster implementation results in faster access to services by communities. Overall, the Time analysis of the SPs in the Study shows mixed results though overall the data shows that more nonKC SPs were completed early or on time compared to KC – and therefore fewer nonKC SPs were overdue.
  • Delays are unfortunately common in all general construction projects regardless of modalities, usage, funding source, etc. There is no standard approach to the recording of time delays.
  • For those SPs which experience excessive delays, there is first no definition of what would constitute an excessive delay and no details found regarding an escalation process to support more timely resolution. In addition to identifying the need for an escalation process, the AARC-WYG team also identified the need to adopt a common measurement of project duration and to codify data regarding the reasons for delay.  Obtaining more specific insights regarding the duration of delays and the reasons for delays will be of great assistance to future teams working to increase on-time delivery.
Overall Quality Findings
  • The overall findings for Quality are mixed. For roads, the quality is generally similar between KC and non-KC.  In schools, the non-KC schools are all better quality than the KC comparators.  In environmental protection, two of the non-KC projects were better quality than KC with the third pair being generally similar.  In the water systems, all three of the KC systems were better quality than non-KC ones with one non-KC SP considered to be poor quality.
  • The collected quality data was insufficient to establish a clear and integrated approach to quality control linking national standards (such as the Philippines Building Code – and International standards too in some cases) with the detailed guidance in respective infrastructure manuals with SP designs with SP contracts with the actual SP build and the final signed off technical inspection and acceptance documentation. Quality related documentation tended to be missing from project documentation of all modalities and all SP types.
  • The AARC-WYG concluded that more technical input is needed in SP design, construction and inspection and acceptance across all modalities and types. This was especially the case when the engineering complexity was relatively more complex.  In the KC SPs, only one of the twelve SPs availed of the Technical Assistance Fund whilst others, from the limited detailed actual cost data, allocated budgets for technical expertise were underspent.
  • Findings relevant to technical input during design include, for example, adaptation of standard school designs to site to resolve factors such as source of electricity supply and drainage. During construction, paving cracking was observed in all road SPs which may have been mitigated with appropriate technical input at the correct point.  Electrical works in both KC and nonKC schools should have required a licensed electrician however the overall quality was poor and with one instance of dangerous work which should not have passed a final electrical inspection or acceptance according to building code standards.
  • These types of deficiencies were found across all types of SP and translated into corresponding quality defects in the constructed SPs.
  • The SP design drawings collected and analyzed in the Study varied in consistency of quality and details. Overall, it was judged that the SP designs did not provide adequate information to assure a good quality of construction by communities or contractors and to clearly direct inspection and acceptance.  This applies equally to all SP types though it was noted that water system designs tended to be better than roads, schools and EP.
  • There were instances when SPs were inspected and accepted despite there being quality related defects. In the final acceptance process, the documentation provided to the AARC-WYG tends to place priority on administrative and financial compliance before technical completion with designs and contracts and quality.  Often the engineer signatory is “noted”.
Overall Cost Findings
  • For six of the 12 SP pairs, the standardized unit cost of the KC SP was lower than the unit cost of the non-KC SP in the pair.
  • The available data shows that the unit cost of management costs was higher for KC SPs than for non-KC SPs, whereas the unit cost of construction was lower. For roads, the higher unit cost of management was offset by substantially lower unit construction costs, thereby resulting in lower total costs per unit. For water systems, the total unit cost was almost the same for KC and non-KC SPs. For school buildings and EP SPs, non-KC SPs enjoyed a cost advantage, mainly because of high KC management costs especially for those SPs which were delayed.
  • Effective financial controls were undermined by the limitations of the available cost data. The available data was not sufficient to establish the financial control links between the planned costs, the contracted costs and the actual costs at item-level detail.
  • Sixteen of the twenty-four SPs were completed below their initially planned cost including eleven of the twelve KC SPs and 5 of the non-KC SPs. The issue in the view of the AARC-WYG Team is that some portion of these unused funds could be spent in increasing the current low level of technical input to the SPs which, as identified in the Quality Related Findings, is an issue impacting on overall SP quality.
Overall Sustainability Findings
  • Overall, SP sustainability appears to be challenged except for KC water systems. In most of the SPs, except for two KC water system SPs, there is evidence for sustained funding for future O&M. An overwhelming majority of the SPs have very little financial support from LGUs.
  • MLGUs and BLGUs tend to be an integral part of local infrastructure development process and, in the case of KC at least, the question appears open on the extent to which existing arrangements should be used compared to the creation of new arrangements such as an O&M committee. Overall, the results appeared mixed in relation to which arrangement is most effective overall in ensuring the adoption of sustainability roles and responsibilities.
  • All communities in both KC and non-KC SP areas have some degree of local technical expertise associated with construction job. There also exists a wider acceptance and practice of community volunteer mobilization for regular O&M work (largely cleaning, grass cutting, removal of extraneous vegetation etc). This helps in general upkeep of SPs along with minor maintenance.
  • It is commendable that the KC program acknowledged the importance of both community participation and maintenance for infrastructure projects. However, it appears that most communities submitted O&M plans that were copies of KC’s source document to meet award requirements, and invested nothing further. This highlights an issue for the capacity development.
  • Unfunded Agreements. Sustainability is a complex issue as highlighted in the previous Section and warrants more research than possible in this study.  Ultimately, the burden for maintenance and repair of the constructed community level infrastructure falls to the LGU regardless of modality.  The respective lead government agencies seek a Memorandum of Agreement (MOA) with the LGU to secure their commitment that they will take care of the infrastructure.
  • The main weakness with the current MOA approach is that the LGU is not bound to ringfence funds – only to accept responsibility for maintenance and repair. This means that the LGUs are most probably committed to a total value of MOAs across all sectors and modalities (significantly?) beyond their financial resources especially for those LGUs in poor areas which have less income.
  • Medium-long-term sustainability. As discussed previously, it is very unlikely that LGUs have the financial capacity and support to generate the funds and other resources required to even resolve dangerous conditions.  Therefore, under the current arrangements and the relative financial strength of the LGUs (as previously stated all the SPs are in poor LGUs with minimal financial resources) sustainability lies in various coordinated actions.  These include better design which promotes sustainability, better quality management to reduce construction deficiencies, improved local skills in the community to undertake preventive maintenance and to fully utilize available funds including specialist technical roles.

Overall Conclusions

Case for Different Modalities Appears Unclear and Unproven
  • This Study encountered five different modalities for building community infrastructure. The main trigger for a different modality and supporting management system appears to be funding source and the attendant prioritization and selection of proposed SPs.  This would appear to have some merit so that respective NGAs are able to ensure the funding approval is aligned with strategic objectives and policy direction
  • What appears less clear is why this should in turn lead to the creation of different management system for actual implementation – and arguably even for the proposal and approval. The result of multiple modalities is duplication and inefficiency of process which can serve to confuse stakeholders trying to understand which SP is in which modality following which management system.  This is costly and means that rather than building a common implementation approach with improved management of the project constraints (time, quality, cost and sustainability) there is arguably a dilution of effort, learning and practice.
 

Lessons Learned/Recommendations:

KC does not appear to have changed the quantity or quality of citizen participation in local governance beyond the project.  This is a key aspect of the KC theory of change, in which communities experiencing the KC process—with its extensive meetings and wide community engagement—apply a similar model to other, non-KC efforts.  The evaluation does not find any evidence that this has occurred.  One explanation for this is that other the current barangay governance structure does not easily allow for KC -like engagement. Citizens can be involved in the Barangay Assemblies, but these only have been twice a year and are mainly for reporting purposes, not engagement about project decisions. The Barangay Development Council, which is more geared towards project selection, is not open to community participation. For KC to affect the quantity or quality of citizen participation in local governance beyond the project, local governance may need to be more inviting of citizen participation.  Local political leaders may need to be targeted for capacity building and/or other interventions as part of the project.  The actual subprojects constructed by KC appear to have created some gains associated with several types of projects.  Completed roads SPs led to improved travel time and costs to basic services and lowered transport costs for agricultural products, although they negatively affected agricultural productivity, and had no effect on productivity for fisheries or livestock and poultry. Education SPs improved enrollment in primary and secondary schools; and water SPs reduced the time and cost spent obtaining water. All gains were important, medium-sized gains.  The overall gains from these diverse SPs remain limited when compared to the costs, largely because of losses in rice production associated with roads SPs.  The project’s overall ERR thus remains relatively low (3%). Those outcomes which truly span SP type (household consumption and assets) and thus apply to the full set of municipalities that had completed SPs also do not indicate any impacts due to KC.  In other words, KC did not achieve long run goals of poverty reduction, as measured by household assets, consumption, warehousing value; or labor force participation or household earnings. However, it may be too ambitious to expect subprojects to translate into changes in consumption or assets over a three-year timeframe. It is hard to say that KC surely promotes better governance or citizen engagement. Some aspects indicating the early stages of KC – participation in local governance, knowledge and awareness of local governance, peer interaction – all improved. If the policy goal is to improve these indicators, then seemingly the CEAC process could be effective at promoting these goals. However, it may be too much to expect that KC is able to transform how citizens engage with and rely on each other and government. Residents did not buy into the KC process so much as to replicate it in other contexts outside of KC, perhaps because government structures do not allow for that, or because the KC process wasn’t yet so ingrained that communities felt the need to apply it in other contexts. In fact, this study shows that being exposed to the KC process decreased residents’ confidence and expectation that they can improve their circumstances. This could be because they became engaged but disillusioned by the process, or that it is challenging from an individual perspective to see how one’s actions affect long-run outcomes. Thus while KC met implementation goals of greater community engagement in the KC process and delivering benefits to residents through subprojects, it fell short of some project expectations, namely transforming how communities interact and engage with government. This evaluation, along with others in the CDD field, supports the questioning of the CDD theory of change, and asks whether projects that deliver basic infrastructure should limit costs and aim to focus only on this infrastructure rather than broader, more transformative processes. From a policy perspective, it is safe to conclude that the KC process is better than the status quo at identifying residents’ public goods preferences. Residents’ needs were clearly better met by KC. In the future, CDD-like processes could be incorporated into programs to select, design and/or site small community infrastructure.  There is less to say about whether KC as an implementation model is superior to other modalities.   Indeed, just over half of survey respondents in KC areas said they thought KC and the municipal or other government implementer were equally good at implementing projects that benefit the community (only a third said that KC was superior). Thus, it seems sensible from a policy perspective to attempt to replicate the early CEAC stages in terms of project prioritization and gathering community input on what projects to implement. It may also be worth further research to test what implementation modality results in superior implementation quality. One can envision that a different implementation model with heavy citizen engagement and input at the outset could still generate projects that met community needs. According to the cost study, the Implementation model that is used should prioritize project quality and sustainability over cost and speed.  It is often argued that achieving time, cost and quality is not possible and that only two of the three are viable – and this has been evident in all SP implementation. The dual reasons for prioritizing quality and sustainability are that the SP may be a community’s first engagement with the Government in this way and therefore, drawing on global CDD lessons, it is critical that communities see a positive result from their participation.  And in the view of the AARC-WYG Team, that should be good quality infrastructure. The other reason for prioritizing quality is to also the need to ensure sustainability.  The pressure on financial resources for infrastructure O&M is well known therefore the lower the O&M burden because of better quality the higher the likelihood of sustainability.  This recommendation recognizes that improving quality in the initial design and construction will enhance sustainability and reduce the whole life-costing of the infrastructure. SP sustainability showed mixed results across all modalities.  Further study may be needed to determine the most effective mix of interventions and activities which maximize community infrastructure sustainability.  Specific Improvements to SP sustainability can be found in the report. Early in project design of community-driven development (CDD) projects, MCC should review if different categories of community members’ work are paid and unpaid, and the effects of these categories on gender dynamics.  Women’s volunteer work in KC was viewed as having many benefits, including raising the status of women in their communities and giving them leadership opportunities and a chance to participate in local decision-making.  

Next Steps

In terms of the evaluation, there will be further analysis of agriculture productivity related to roads projects to try to understand the counter-intuitive finding that roads projects reduced agricultural productivity.