The MCC compact with the Philippines was a five-year investment of $385 million. The $125 million Kalahi-CIDSS (KC) Project is the subject of an impact evaluation and a cost study, written up in two reports: “Kalahi-CIDSS Impact Evaluation Third Round Report (Impact Evaluation)” and “KALAHI-CIDSS Subprojects Cost Study (Cost Study.)” The KC community-driven development project sought to improve residents’ socioeconomic situations, help them become more engaged in local government, and empower communities. By compact end, KC had financed more than 4,000 small infrastructure projects (including school classrooms, water systems, farm-to-market roads, etc.) and trained community volunteers in more than 3,000 villages.
The Impact Evaluation, a five-year randomized control trial with three rounds of surveying paired with real-world observations and qualitative research, found both marked successes and notable areas where no effects were detected. Projects implemented in KC areas, even those not funded by KC, were more reflective of residents’ stated priorities than projects implemented in non-KC areas. Consistent with this finding, residents’ satisfaction with the program was extremely high. Services like roads, education, and water delivered benefits to residents like reduced travel time and costs, reduced agriculture transport costs, increased enrollment, and reduced time and costs to obtain water. In contrast and unexpectedly, roads projects reduced agricultural productivity. This finding will be the subject of further research by the evaluator. Lastly, KC was not as effective at generating broader, post-project social changes related to improved governance or community empowerment.
The Cost Study assessed the cost, quality, time frame and sustainability of the infrastructure built via community-driven development (KC) and centrally planned projects (non-KC) that were similar in type and scale. In terms of cost, for both KC and non-KC actual unit costs were lower than planned. For quality, KC water projects were of higher quality than non-KC ones, while KC roads were of similar quality, and KC schools were of lower quality. Overall, the time analysis shows that more non-KC projects were completed early or on time compared to KC. In terms of sustainability, in most of the sub projects, except the KC water systems, there is little evidence of funding for future operations and maintenance (O&M).
The lessons learned are: The participatory KC process is better than the status quo at identifying residents’ small infrastructure preferences. In the future, CDD-like processes could be incorporated into programs to select, design and/or site small community infrastructure.
KC does not appear to have changed citizen participation in local governance beyond the project. If this is to remain a key aspect of the CDD theory of change, local political leaders may need to be targeted for capacity building or other interventions as part of the project.
Results were analyzed for different subgroups, women versus men, Indigenous Peoples versus Non-Indigenous Peoples and poor versus non-poor. The only cases in which KC affected people differently was for Indigenous Peoples (IPs). For example, IPs appear to benefit substantially more from improvements in access to education than do non-IPs .
The quality and sustainability of community infrastructure, built via KC or other government ministries, should be prioritized over cost and time. Local design guidelines should be considered along with international standards to determine the appropriate design. Coupling quality and sustainability, is important, because O&M expenditures are often not financed. Therefore, the lower the O&M burden because of better quality infrastructure, the higher the likelihood of sustainability.
It may be worth further research to test what implementation modality results in superior infrastructure quality. One can envision that a non-community-driven-development implementation model with heavy citizen engagement and input at the outset could still generate projects that meet community needs.
In terms of the evaluation, there will be further analysis of agriculture productivity related to roads projects to try to understand the counter-intuitive finding that roads projects reduced agricultural productivity.
In ContextThe MCC compact with the Philippines was a five-year investment from 2011-2016 of $385 million in 3 projects: the Kalahi-CIDSS Project (KC), the Revenue Administration Reform Project (RARP) and the Secondary National Roads Development Project (SNRDP). The Kalahi-CIDSS (KC) Project is a nationwide, government-run, community-driven development (CDD) project in the Philippines. KC pairs community training with block grants at the barangay or village level, which are meant to enable communities to address their self-identified development needs, largely through financing and building public infrastructure and public services called “subprojects.” The KC project was financed by both MCC ($125 million) and the World Bank ($59 million.) The KC Project is the subject of two independent evaluations; the results of these studies are summarized here. The KC component represents 32 percent of the total compact. Other components of the compact are the subject of independent evaluations. *These figures are based on MCC obligations as of September 2016
Program LogicThe Philippines lagged significantly behind other countries in the region with respect to government development expenditures as a percentage of GDP and infrastructure investment and quality. The Asian Development Bank’s 2007 growth diagnostic report found that inadequacies in infrastructure were a critical constraint to growth and that the availability of basic infrastructure (water, sanitation, roads, electricity) was regressive. Provision and use of education and health services were found to vary across regions, particularly as a function of incomes. Community-driven development projects are a strategy for addressing these constraints and providing community empowerment and poverty reduction. In the past, they have been used to support a wide range of community priority needs including provision of water supply and nutrition programs for women and children; building of school classrooms, day care and health facilities, farm to market roads, foot bridges, and drainage systems; and support for productive enterprises such as pre- and post-harvest facilities as well as community capacity building. Through KC, communities (“barangays” or villages), together with their village and municipal governments, were trained, to choose, design and implement sub-projects that were intended to address communities’ most pressing needs. This was done through a three-year, three-cycle program, which included “social preparation” training for communities, barangays, and municipalities, and sub-project implementation. To address gender concerns, the KC project included dedicated gender staff positions and gender-focused activities, including the provision of “gender incentive grants” to communities. The KC project funded by MCC was an expansion of an initial KC project ensure (“KC1”) that was implemented between 2003 and 2010. KC1 was funded by a loan from the World Bank. The KC Project included the following activities.
- Capacity building and implementation support activities: Millennium Challenge Account-Philippines (MCA-P) provided the staff salaries, logistical support and training for the frontline workers, known as the area coordinating teams (ACTs) at the Department of Social Welfare and Development (DSWD.) The role of the ACTs was to carry out the “Community Empowerment Activity Cycle” (CEAC), each year gradually “handing off” the CEAC to local government. In each year or cycle, barangays held a series of meetings that were facilitated by members of the ACT in which barangay residents identified and prioritized constraints to economic activities within their communities and then identified and prioritized solutions to these constraints. Finally, the barangay presented its solution or subproject to the “Municipal Inter-Barangay Forum” (MIBF) for possible funding. After each cycle or year, there was a transition and reporting period.
- Grants for community projects activity: MCA-P funded DSWD grants to implement community-chosen sub-projects. A Gender Incentive Grant initiative was also funded to foster women’s participation in local development. In addition, the municipalities and barangays provided cash and/or in-kind contributions (including partially-paid labor and local materials) to the sub-projects equal to at least 30 percent of the total sub-project costs. The amount of funding was allocated to municipalities based on the number of barangay within that municipality.
- Project management activity: DSWD received salaries, logistics and training for DSWD project management staff at the regional and national level.
Measuring ResultsMCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes. When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.
Monitoring ResultsThe following table summarizes performance on output, outcome and objective indicators specific to the evaluated program.
|Indicators||Level||Baseline (2011)||Actual Achieved||Target||Percent Complete|
|Activity 1: Capacity Building and Implementation Support|
|Percentage of MLGUs that provide technical assistance in KC sub-project preparation, implementation, and monitoring, based on MOA||Outcome||NA||100||80||125%|
|Percentage of Indigenous persons (IPs) that are present during Barangay Assemblies||Outcome||NA||7.35||No target||No Target|
|Percentage of municipalities that provide their KC Local Counterpart Contributions (LCC) based on their LCC delivery plan||Outcome||NA||100||80||125%|
|Number of barangays that have completed all the trainings during the social preparation stage||Output||0||3,760||3000||125%|
|Activity 2: Grants for Community Projects|
|Number of Subprojects (SPs) with 100% physical accomplishment||Output||0||4,011||3217||125%|
|Number of barangays that have completed specific training on subproject management and implementation||Output||0||3,114||1500||208%|
|Number of Gender Incentive Grant (GIG)-funded Subprojects (SPs)||Output||0||55||No target||No Target|
- Evaluation Section: For the first study, “Kalahi-CIDSS Impact Evaluation Third Round Report (Impact Evaluation)”
Evaluation QuestionsThe evaluation was designed to answer the following questions in three spheres:
- To what extent did
- Subprojects (SPs) improve access to related key services?
- Roads SPs reduce agriculture, fisheries and livestock transport costs?
- Roads SPs improve productivity in agriculture, fisheries, and livestock sectors?
- School SPs increase school enrollment and improve student/ teacher ratios?
- Water SPs reduce time and costs spent obtaining water?
- KC raise household consumption and asset holdings?
- KC raise household labor force participation and earnings?
- To what extent did
- KC increase quantity and quality of participation in local governance around decision-making and implementation related to KC activities?
- KC increase participation in and knowledge of formal structures beyond KC?
- KC improve barangay information sharing and inclusiveness beyond KC?
- KC increase confidence and self-efficacy beyond KC?
- KC increase knowledge and awareness of local governance?
- KC improve degree to which barangay projects correspond to ex-ante preferences?
- KC improve perceptions of local governance?
- KC raise capacity of barangay government?
- Community empowerment
- To what extent did
- KC increase interactions among peers?
- KC increase participation in community organizations?
- KC improve how well communities deal with natural disasters and other hardships?
- To what extent did
- To what extent did
- To what extent did
Evaluation FindingsResults are analyzed across three domains: socioeconomic, institutional, and community empowerment. Each domain includes multiple hypotheses. Each hypothesis makes up an index or group of indicators. These indices reduce the number of unique chance outcomes, since if the hundreds of unique outcomes were examined, some would be statistically significant due to chance. Results for each hypothesis are presented in standard deviation units. An overview of results is shown in the table below. The table includes the regression coefficient in standard deviation units and statistical significance.
|Evaluator||Innovations for Poverty Action|
|Impact or Performance?||Impact|
|Methodology||Randomized Control Trial|
|Evaluation Period||The findings reflect data collected between 2011 and 2015 through a variety of methods, including extensive interviews of nearly 6,000 households, barangay (village) leaders and project staff. Baseline data collection took place, April-June 2012, interim data collection February-June 2014, and third round data collection July-October 2015. Qualitative research complemented the quantitative analysis at baseline and endline to deepen the findings. The study also conducted structured community activities to observe whether KC practices were carried over into other areas of local governance. The Compact ended in May 2016.|
|Outcomes||See text below for interpretation of findings from table
|Effect on household income attributable to MCC||The study found no effect on household income or consumption, but it may be too early to measure this. The study did find statistically significant changes on intermediate outcomes like travel times, and calculated a post-project ERR of 3%, based on these intermediate outcomes.|
- Evaluation Section: For the second study, “KALAHI-CIDSS Subprojects Cost Study (Cost Study.)”
Evaluation QuestionsThe main objective of this cost study was to assess the cost, quality, construction time and sustainability of the infrastructure built via community-driven development (KALAHI-CIDSS – KC) and centrally planned projects (non-KC) that are similar in type and scale. The overarching question was: Do community-driven development (CDD) projects deliver similar quality infrastructure as centrally-planned infrastructure faster with lower cost, and superior sustainability? The evaluation compared similar projects (schools, water systems, roads and environmental protection) implemented either by KC or other government ministries. Below are the main questions.
- What is the difference in construction time for KC and non-KC funded community infrastructure?
- What is the difference in quality for KC and non-KC funded community infrastructure?
- What is the difference in cost for KC and non-KC funded community infrastructure?
- What is the difference in sustainability for KC and non-KC funded community infrastructure?
Evaluation FindingsThis evaluation is a cost study and compared similar projects (education, water, farm-to-market roads and environmental mitigation) implemented by KC and other government ministries. The evaluators relied on document review, information from the relevant ministries’ information systems, meetings with stakeholders and site visits to the infrastructure.
- It was evident that the communities are benefitting from increased access to services provided by the implemented SPs. Overall, each infrastructure was being used by the community in remote rural communities.
All SPs Encountered Inconsistent Implementation
- It is clear from the Study that all SPs are challenged by at least one of the four factors, time, quality, cost and sustainability and usually more than one. This reinforces a widely recognized principle that achieving only two is realistic.
- The resulting impact of the project constraints is that SP implementation is inconsistent across all modalities (modalities include National Government Agency, Bottom-Up Budgeting, Performance Challenge Fund, Local Government Unit and KC). Some SPs may be average quality but with a high likelihood of sustainability or overdue on time but on or under budget. The extent of these inconsistencies suggests scope for improvements so that SP implementation on a nationwide basis can be strengthened. Given that the project constraints are encountered by both modalities (KC and nonKC) in this Study, the inconsistencies suggest that prioritizing two factors or project constraints for improvement would yield results.
Overall Time Findings
- The time to implement a SP is a critical factor as faster implementation results in faster access to services by communities. Overall, the Time analysis of the SPs in the Study shows mixed results though overall the data shows that more nonKC SPs were completed early or on time compared to KC – and therefore fewer nonKC SPs were overdue.
- Delays are unfortunately common in all general construction projects regardless of modalities, usage, funding source, etc. There is no standard approach to the recording of time delays.
- For those SPs which experience excessive delays, there is first no definition of what would constitute an excessive delay and no details found regarding an escalation process to support more timely resolution. In addition to identifying the need for an escalation process, the AARC-WYG team also identified the need to adopt a common measurement of project duration and to codify data regarding the reasons for delay. Obtaining more specific insights regarding the duration of delays and the reasons for delays will be of great assistance to future teams working to increase on-time delivery.
Overall Quality Findings
- The overall findings for Quality are mixed. For roads, the quality is generally similar between KC and non-KC. In schools, the non-KC schools are all better quality than the KC comparators. In environmental protection, two of the non-KC projects were better quality than KC with the third pair being generally similar. In the water systems, all three of the KC systems were better quality than non-KC ones with one non-KC SP considered to be poor quality.
- The collected quality data was insufficient to establish a clear and integrated approach to quality control linking national standards (such as the Philippines Building Code – and International standards too in some cases) with the detailed guidance in respective infrastructure manuals with SP designs with SP contracts with the actual SP build and the final signed off technical inspection and acceptance documentation. Quality related documentation tended to be missing from project documentation of all modalities and all SP types.
- The AARC-WYG concluded that more technical input is needed in SP design, construction and inspection and acceptance across all modalities and types. This was especially the case when the engineering complexity was relatively more complex. In the KC SPs, only one of the twelve SPs availed of the Technical Assistance Fund whilst others, from the limited detailed actual cost data, allocated budgets for technical expertise were underspent.
- Findings relevant to technical input during design include, for example, adaptation of standard school designs to site to resolve factors such as source of electricity supply and drainage. During construction, paving cracking was observed in all road SPs which may have been mitigated with appropriate technical input at the correct point. Electrical works in both KC and nonKC schools should have required a licensed electrician however the overall quality was poor and with one instance of dangerous work which should not have passed a final electrical inspection or acceptance according to building code standards.
- These types of deficiencies were found across all types of SP and translated into corresponding quality defects in the constructed SPs.
- The SP design drawings collected and analyzed in the Study varied in consistency of quality and details. Overall, it was judged that the SP designs did not provide adequate information to assure a good quality of construction by communities or contractors and to clearly direct inspection and acceptance. This applies equally to all SP types though it was noted that water system designs tended to be better than roads, schools and EP.
- There were instances when SPs were inspected and accepted despite there being quality related defects. In the final acceptance process, the documentation provided to the AARC-WYG tends to place priority on administrative and financial compliance before technical completion with designs and contracts and quality. Often the engineer signatory is “noted”.
Overall Cost Findings
- For six of the 12 SP pairs, the standardized unit cost of the KC SP was lower than the unit cost of the non-KC SP in the pair.
- The available data shows that the unit cost of management costs was higher for KC SPs than for non-KC SPs, whereas the unit cost of construction was lower. For roads, the higher unit cost of management was offset by substantially lower unit construction costs, thereby resulting in lower total costs per unit. For water systems, the total unit cost was almost the same for KC and non-KC SPs. For school buildings and EP SPs, non-KC SPs enjoyed a cost advantage, mainly because of high KC management costs especially for those SPs which were delayed.
- Effective financial controls were undermined by the limitations of the available cost data. The available data was not sufficient to establish the financial control links between the planned costs, the contracted costs and the actual costs at item-level detail.
- Sixteen of the twenty-four SPs were completed below their initially planned cost including eleven of the twelve KC SPs and 5 of the non-KC SPs. The issue in the view of the AARC-WYG Team is that some portion of these unused funds could be spent in increasing the current low level of technical input to the SPs which, as identified in the Quality Related Findings, is an issue impacting on overall SP quality.
Overall Sustainability Findings
- Overall, SP sustainability appears to be challenged except for KC water systems. In most of the SPs, except for two KC water system SPs, there is evidence for sustained funding for future O&M. An overwhelming majority of the SPs have very little financial support from LGUs.
- MLGUs and BLGUs tend to be an integral part of local infrastructure development process and, in the case of KC at least, the question appears open on the extent to which existing arrangements should be used compared to the creation of new arrangements such as an O&M committee. Overall, the results appeared mixed in relation to which arrangement is most effective overall in ensuring the adoption of sustainability roles and responsibilities.
- All communities in both KC and non-KC SP areas have some degree of local technical expertise associated with construction job. There also exists a wider acceptance and practice of community volunteer mobilization for regular O&M work (largely cleaning, grass cutting, removal of extraneous vegetation etc). This helps in general upkeep of SPs along with minor maintenance.
- It is commendable that the KC program acknowledged the importance of both community participation and maintenance for infrastructure projects. However, it appears that most communities submitted O&M plans that were copies of KC’s source document to meet award requirements, and invested nothing further. This highlights an issue for the capacity development.
- Unfunded Agreements. Sustainability is a complex issue as highlighted in the previous Section and warrants more research than possible in this study. Ultimately, the burden for maintenance and repair of the constructed community level infrastructure falls to the LGU regardless of modality. The respective lead government agencies seek a Memorandum of Agreement (MOA) with the LGU to secure their commitment that they will take care of the infrastructure.
- The main weakness with the current MOA approach is that the LGU is not bound to ringfence funds – only to accept responsibility for maintenance and repair. This means that the LGUs are most probably committed to a total value of MOAs across all sectors and modalities (significantly?) beyond their financial resources especially for those LGUs in poor areas which have less income.
- Medium-long-term sustainability. As discussed previously, it is very unlikely that LGUs have the financial capacity and support to generate the funds and other resources required to even resolve dangerous conditions. Therefore, under the current arrangements and the relative financial strength of the LGUs (as previously stated all the SPs are in poor LGUs with minimal financial resources) sustainability lies in various coordinated actions. These include better design which promotes sustainability, better quality management to reduce construction deficiencies, improved local skills in the community to undertake preventive maintenance and to fully utilize available funds including specialist technical roles.
Case for Different Modalities Appears Unclear and Unproven
- This Study encountered five different modalities for building community infrastructure. The main trigger for a different modality and supporting management system appears to be funding source and the attendant prioritization and selection of proposed SPs. This would appear to have some merit so that respective NGAs are able to ensure the funding approval is aligned with strategic objectives and policy direction
- What appears less clear is why this should in turn lead to the creation of different management system for actual implementation – and arguably even for the proposal and approval. The result of multiple modalities is duplication and inefficiency of process which can serve to confuse stakeholders trying to understand which SP is in which modality following which management system. This is costly and means that rather than building a common implementation approach with improved management of the project constraints (time, quality, cost and sustainability) there is arguably a dilution of effort, learning and practice.