The MCC compact with Malawi is a five-year investment (2013-2018) of $350.7 million. The $26.5 million Power Sector Reform Project component is the subject of an independent performance evaluation summarized here.
- The objective of the Power Sector Reform Project is to create an enabling environment for future expansion of the power sector by strengthening pivotal sector institutions and enhancing regulation and governance of the sector, by rebuilding the Electricity Supply Corporation of Malawi (ESCOM) into a financially strong, well-managed utility and developing a regulatory environment that supports public and private investment in new generation capacity and expanded access. Project achievements to date include support for ESCOM’s 2014 tariff application leading to an increase in electricity tariffs, and approval by the Government of Malawi of a roadmap for the restructuring of the power market to attract private investment.
- The evaluation anticipates three rounds of data collection, the first of which is the subject of this summary. An assessment of the first year of project implementation suggested that there is a need to cultivate champions among project partners and other Compact beneficiaries. Moreover, a mechanism is required for key stakeholders to ensure coordination and executive management decision-making on reform recommendations and implementation. Consultancies should be better prioritized and sequenced to avoid overwhelming project partners, particularly ESCOM.
- Key lessons learned: reform programs require significant and continued commitment from host-country partner organizations to succeed. In addition, reform projects encompass a broad set of interventions that do not always fit neatly into the traditional project management framework, making it essential to work outside the bounds of specific technical assistance activities to conduct continuous stakeholder engagement. Lastly, host-country governments and program partners may require incentives throughout the Compact period in order to implement difficult reforms, rather than only prior to Entry into Force.
- The second round of data collection for the evaluation occurred in November 2016 and the associated reports will be released in 2018. The final round is expected to occur in 2018.
The Millennium Challenge Corporation (MCC) compact with Malawi is a five-year investment (2013-2018) of $350.7 million in 3 projects: the Infrastructure Development Project, the Power Sector Reform Project, the Environmental and Natural Resources Management Project. The Power Sector Reform Project (“PSRP”) includes 2 major activities: the ESCOM Turnaround Activity and the Regulatory Strengthening Activity. The $26.5 million PSRP component is the subject of the results described here. This component is equivalent to 7.6% of the total Compact investment. 1 Other components of the compact are the subject of forthcoming independent evaluations.
The Power Sector Reform Project was designed to address the financial, operational, and governance challenges among power sector institutions by providing support for the Government of Malawi’s policy reform agenda. The objective of the PSRP is to create an enabling environment for future expansion of the power sector by strengthening pivotal sector institutions and enhancing regulation and governance of the sector, by rebuilding the Electricity Supply Corporation of Malawi (ESCOM) into a financially strong, well-managed utility and developing a regulatory environment that supports public and private investment in new generation capacity and expanded access. The project will also build capacity and provide technical assistance to the Malawi Energy Regulatory Authority (MERA) and the Ministry of Natural Resources, Energy and Mining.
MCC and Millennium Challenge Account-Malawi (MCA-M) expect that the investments in reforming the power sector, combined with other Compact investments to upgrade and rehabilitate power infrastructure, will ultimately lead to a reduced cost of doing business in Malawi, expanded access to electricity for Malawian people and businesses, and increased value-added production in Malawi.
There were several key assumptions underlying the PSRP program logic during the design of the investment. The salient assumptions include:
- Political will exists to implement and sustain reforms, particularly in processes such as procurement.
- Political will exists to permit an increase in tariffs to cost-reflective levels.
- The ESCOM Board commits to new organizational structures and human resources (HR) practices.
- Technical staff turnover within ESCOM and MCA is minimized.
- Parliament approves necessary reforms that permit an enabling environment.
For a more detailed description of the reform project, please refer to page 10 of the Malawi M&E Plan, which can be found here: https://assets.mcc.gov/content/uploads/FINAL-ME-Plan-V3.pdf
MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes. When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.
The following table summarizes performance on output and outcome indicators specific to the PSRP.
|Indicators||Level||Baseline (2012-2013)||Actual Achieved (03/2015)||End of Compact Target
|Operating cost-recovery ratio (based on operating expenses) (Percentage)||Outcome||161||155.4||150||3.6% 2|
|Operating cost-recovery ratio (based on operating expenses + depreciation) (Percentage)||Outcome||150||160||128||25% 2|
|Operating cost-recovery ratio – operating expenses + depreciation + return (weighted average cost of capital (WACC) X rate base). (Percentage)||Outcome||113||101.7||100||1.7% 2|
|Current Ratio||Outcome||6.48||2.2||2.0 – 4.0|
|ESCOM Turnaround Activity|
|Average Collection Period – Annual (Days)||Outcome||72||75||60||25%2|
|Average Collection Period – Quarterly (Days)||Outcome||72||98||60||63%2|
|Average Cost of Electricity Billed (US$/kWh)||Outcome||0.07||0.10||No Target||N/A|
|Bad Debt (Percentage)||Outcome||25||48||2||2300%2*|
|Average Creditor Days – Annual (Days)||Outcome||27||156||45||247%2**|
|Average Creditor Days – Quarterly (Days)||Outcome||27||45||45||0%2|
|ESCOM Maintenance Expenditures ratio to planned maintenance budget (Percentage)||Output||128||33.41||100||66.6%2|
|Transition to Pre-paid metering system (Percentage)||Output||36||60.9||100||39%|
|Regulatory Strengthening Activity|
|Average tariff per kilowatt-hour (USD)||Output||.08||.09||.13||20%|
Source: September 2017 ITT, based reporting from ESCOM, the Ministry of Energy, the Ministry of Finance, MERA, and MCA-Malawi.
Notes on indicators:
* This indicator measures the percentage of accounts receivable that are over 90 days past due and provide an indication of financial losses due to uncollectable debt. Bad debt increases beyond baseline level are likely to have been due to increasing tariffs and electricity sales, creating additional collection burden, and substantial long-standing Government of Malawi arrears.
** This indicator measures how long it takes a company, on average, to pay its creditors, providing insight into a company’s creditworthiness from a supplier’s perspective. Performance on this indicator is based on the value reported at the end of Year 1 of the compact and represents significant deterioration from baseline, although compact interventions would not necessarily be expected to have an effect at this stage.
The average completion rate of output targets is 11 percent and targets were met or exceeded in 0 of the 20 output indicators. The average deviation rate of outcome targets is 273.5 percent and targets were met in 0 of the 10 outcome indicators. 3 However, given that many outcomes used to measure sector performance are complex ratios and benchmarks designed to remain within an optimal range, it is difficult to interpret results in terms of linear progress towards “completion.” Therefore, each outcome should be interpreted individually as a reflection of the financial and operational sustainability of the power sector. Also, please note that the data above reflect project results at a relatively early stage of Compact implementation.
The evaluation seeks to answer the following questions for each of the PSRP consultancies:
- What is the quality of PSRP consultant deliverables?
- How is the working relationship between PSRP consultants and their counterparts in partner institutions?
- What are lessons learned from the PSRP consultancies?
In addition to internal monitoring of the project, MCC commissioned a performance evaluation to assess the results of the PSRP Project. It consists of several complementary approaches, one of which entails a process evaluation, or assessment of the quality and fidelity of project implementation. Other components of the evaluation include workflow studies focused on ESCOM operations, key informant interviews, and other qualitative data collection. The evaluation planned for three rounds of data collection, and this summary presents the findings of the first round of the process evaluation. The findings of the first round from other evaluation approaches are not presented as they establish a baseline and do not reflect the implementation or results of the project. The process evaluation includes a review of documentation (e.g., terms of reference (TORs), inception reports, and reporting documentation) for each of the consultancies, a short mini-survey to MCC and MCA-M personnel involved in the Malawi Compact, and key informant interviews with the consultants and their points of contacts among the various Project Partners involved in implementing the Compact. 4
|Evaluator||Social Impact, Inc.|
|Evaluation type||Performance evaluation|
|Methodology||Process evaluation (assessment of implementation)|
|Evaluation Period||Program implementation: Sept 2013 – present (will continue through Sept 2018)
Data collection: February-March 2015
|Outcomes||PSRP outcomes will be the subject of future evaluation reports.|
|Objective-level Outcomes||PSRP objective-level outcomes will the subject of future evaluation report.|
|Process Evaluation Findings||Six broad themes on the challenges and lessons learned through the data collection field work and data analysis are highlighted below.
· The PSRP design did not sufficiently prioritize and sequence consultancies, so beneficiaries felt overwhelmed: With so many consultancies at one time, ESCOM has become overwhelmed and has not been able to act on many of the consultancies’ recommendations. Management has been unable to balance Compact demands on top of the already considerable challenges of leading the day-to-day ESCOM operations. In hindsight, some of the consultancies could have been deprioritized and delayed (e.g., the tariff consultancy, organizational review, and social and gender audit).
· ESCOM has not owned the PSRP: There is a general sense that ESCOM does not have a sufficient degree of ownership over the PSRP despite MCC and MCA-M efforts to engage ESCOM in consultancy TOR development. There is no full-time PSRP coordinator within ESCOM, and points of contact are at times not clear, too high-level, and must balance Compact consultants on top of their already demanding full-time workloads. As a result, points of contact have been unable to adequately support consultants, ensure that deliverables respond to ESCOM’s needs, and push forward management decision-making and subsequent implementation.
· There is no mechanism to ensure ESCOM coordination and executive management decision-making on PSRP recommendations and implementation: Given the demands on executive management’s time and the number of PSRP consultancies, there is a strong need for mechanisms for coordinating, reviewing recommendations, making decisions regarding implementation, and developing an action plan and calendar for implementation. However, no such model currently exists. MERA offers a model in this regard. The regulator established a Working Committee comprised of senior officers below the Director level to facilitate and support the Aurecon consultancy.
· The more successful consultancies have reduced cultural differences and built ownership through a bottom-up, needs-based approach: There are many processes within ESCOM that need to be modified or improved; however, there are often rational reasons for an unsatisfactory status quo. Some consultancies might have made the mistake of starting with the ideal – how things should be done – rather than starting with where ESCOM currently is and building out from there. Many ESCOM personnel have been relatively cold to consultancies they perceive to be telling them how to do their job. By contrast, ESCOM and GOM personnel have shown a strong appreciation for the bottom-up needs-based approach of the MIS consultancy, the Aurecon consultancy, and the High-Level Energy Advisor. There are limits to generalizing based on a few examples, as the MIS is a high priority and good relations do not necessarily ensure impact.
· The PSRP has focused on outputs over outcomes and recommendations over implementation: In order to ensure accountability, PSRP consultants are asked to produce a number of written deliverables. Unfortunately, this has created a focus on outputs over outcomes. While well-developed reports and recommendations are extremely important, they risk being shelved and not used. As many of the consultancies are relatively short term, they end too soon to oversee implementation. The insufficient focus on implementation is a major risk to the PSRP.
· MCA-M’s role has focused on contract management rather than reform promotion: MCC and MCA-M are both based in Lilongwe; however, most of the PSRP activities (as of data collection) had occurred in Blantyre, where ESCOM is headquartered. Given ESCOM’s lack of ownership and the short nature of many consultancies, there is no mechanism to push reforms within ESCOM forward.
|Effect on household income attributable to MCC||This outcome is not measured by this evaluation, given its limited scope.|
There were several key lessons learned from this evaluation for MCC and future partner countries to consider when designing and implementing similar reform projects and evaluations:
- Reform programs require significant and continued commitment from host-country partner organizations to succeed. MCC must put in substantial, up front work to develop a collaborative relationship with program partners, and program partners must establish appropriate internal practices (e.g., change management processes, accountability mechanisms, project management structures) to actively participate in and benefit from programming targeting policy and institutional reform.
- Host-country governments and program partners may require incentives in order to implement difficult reforms throughout the Compact period. In Malawi, Compact entry-into-force was used as an incentive for the Government to achieve several key reforms, including the restructuring of debt at ESCOM to clear its balance sheet. However, the Compact was not explicitly designed with similar incentives and accountability mechanisms that could be used during implementation.
- In many cases, program partners will have capacity constraints and require significant, long-term support to achieve expected substantive outcomes. Therefore, reform programs must be designed so that activities provide sufficient support (in duration and level of effort) to program partners to achieve concrete outcomes rather than simply outputs (such as reports) or recommendations which may not be implemented. This significant resource requirement must be accurately reflected in implementer terms of reference (TORs) and contracts.
- Technical assistance and capacity building work must be planned to align with opportunities for program partners to actively apply new knowledge and skills. MCA-M and MCC are looking for opportunities in the last two years of the Compact where technical assistance activities can be targeted to align with ongoing sector initiatives (e.g., tariff reviews) to provide learning-by-doing opportunities.
- To be successful, reform programs require continuous stakeholder engagement by both MCA and MCC staff (including the Resident Country Mission), particularly as many critical reform interventions do not fit neatly within the bounds of specific technical assistance activities. Engaging with the general public (including citizen groups) is also important to generate support for Compact initiatives and to avoid creating backlash where Compact activities are not well understood. These efforts should ideally be led by the host-country stakeholders but with Compact encouragement.
- The success of reform programs depends on changing practices, reshaping accountability mechanisms, and building new skills and knowledge. These are difficult to achieve and require significant day-to-day effort and coordination on the part of program partners, consultants, MCA and MCC. It is essential to put in place project management and communication practices early on to coordinate these challenging efforts.
The findings presented above reflect the first round of the process evaluation, which is part of the broader PSRP Performance Evaluation. Another round of data collection occurred in November 2016 and the associated reports will be released in 2018. A third and final round of data collection is planned for 2018 and the final evaluation report is expected to be released to the public in 2019.