Introduction

In 2006, the Millennium Challenge Corporation (MCC) and the Government of Armenia (GoA) signed a five-year, $235.65 million compact to improve two main barriers to economic growth: a lack of reliable infrastructure and the slow development of businesses, particularly agribusinesses. The compact consisted of two projects:

• The Irrigated Agriculture Project ($145.67 million) aimed to address the physical, managerial, and financial investments needed to generate sustainable increases in rural income through irrigated agriculture. This project consisted of two complementary activities: (i) the Infrastructure Activity and (ii) the Water-to-Market Activity (WtM).
• The Rural Road Rehabilitation Project (RRRP) ($67.1 million) aimed to rehabilitate up to 943 km of the high-priority lifeline network, a network of secondary and local roads which sought to ensure that all communities, towns and villages were linked to the main road network, either directly or through other communities.

Through the investment, GoA and MCC expected approximately 426,000 people to benefit over 20 years.

MCC’s Board of Directors (Board) selected Armenia as eligible to develop a compact in May 2004. The compact was signed on March 27, 2006 and entered into force on September 29, 2006. In May 2008, MCC’s CEO placed a hold on the disbursement of all remaining funding for the RRRP (approximately $58.65 million) due to  actions by the GoA following the February 2008 presidential elections that MCC determined to be inconsistent with its eligibility criteria, particularly with regard to democratic governance. At its June 2009 meeting, MCC’s Board endorsed the hold, directing MCC not to resume funding for the RRRP for the duration of the compact due to these actions and Armenia’s lack of demonstrated commitment to MCC’s eligibility criteria. This effectively ended all remaining planned MCC funding and work for the compact’s road project through compact closure and reduced the original $235 million compact amount to approximately $177 million.

Upon compact completion on September 29, 2011, the GoA spent 99.7 percent of the $177 million in remaining compact funding.

Under the Irrigated Agriculture Project, 17 irrigation pumping stations were rehabilitated; over 45,000 farmers were trained in on-farm water management techniques; over 35,000 farmers were trained in high-value agriculture (HVA) techniques; five gravity-fed irrigation systems were constructed, including almost 50 kilometers of canals; over 260 water structures were installed on six main canals; approximately 230 kilometers of tertiary canals were rehabilitated and improved across approximately 100 communities; and $12.7 million of credit to over 1,000 farmers was distributed through existing Armenian banks and other financial institutions. The independent evaluation found that while farmers near tertiary canals perceived improvements in the timeliness and reliability of irrigation water and some simple high-value practices such as soil preparation were adopted, key outcomes were not realized such as irrigating more land, producing more HVA crops, or increasing household income.

Under the Rural Road Rehabilitation Project, the compact originally aimed to improve 943 km of roads. After the project re-scoping in April 2008 due to currency devaluation and construction cost escalation, the target was reduced to 332 km. As a result of the May 2008 hold on funding due to post-election events in March 2008, only 3 percent of the original target was completed using compact funds. However, the World Bank used the MCA-funded detailed designs to construct or rehabilitate approximately 150 km of additional rural roads and the GoA decided to co-fund the project in the amount of USD $16.8 million to rehabilitate certain road sections. Before MCC placed the hold on funding, 24.4 kilometers of road in rural Armenia were rehabilitated, road roughness improved by 75 percent, and average daily traffic increased by 15 percent. An independent evaluation covering the roads designed by MCC, but built with World Bank funds, found that the road rehabilitation efforts improved road quality and increased the use of roads in the short term. However, those changes were not sufficient to stimulate increases in agricultural production and sales in a time frame of one to two years.

This report provides a summary of the tangible outputs of the compact program, documents changes in compact activities and the reasons behind them, details information on performance against targets in the monitoring plan, and summarizes the results of the independent evaluations.

Country Context

A small, landlocked country in the Caucasus region, Armenia struggled to recover from the severe economic setbacks caused by the collapse of the Soviet Union in the early 1990s. In 1994, Armenia adopted a comprehensive stabilization and reform program that transformed it into a liberal market economy and launched a period of growth that saw its GDP increase at an average annual rate of 8 percent over the next decade.

This period of growth disproportionately benefited inhabitants of the capital, Yerevan, while the rural poverty rate remained above 30 percent. More than one million Armenians, or about 35 percent of the population, lived in rural areas and were dependent on semi-subsistence agriculture at the time of compact development. These farmers were operating on small, fragmented parcels of land and were constrained by poor infrastructure and an underdeveloped agricultural economy.

At a Glance

• Original Amount at Compact Signing:
  $235,650,000
• Revised Compact Amount:
  177,000,000
• Amount spent:
  176,550,000

• Signed:
  March 27, 200
• Entry Into Force:
  September 29, 2006
• Closed:
  September 29, 2011

• 427,623Estimated beneficiaries over 20 years
• $150,440,000Estimated net benefits over 20 years

  Estimated benefits correspond to $176.55 million of compact funds, where cost-benefit analysis was conducted. Net benefits refer to discounted benefits minus discounted costs.

• M&E Plan
• Post-Compact M&E Plan
• Key Performance Indicators
• Compact Agreement

Compact Development Process

As one of MCC’s first partner countries selected by the Board as eligible to develop a compact in May 2004, the Armenia Compact predates MCC’s use of the constraints analysis in the compact development process. 1 As such, the GoA based its program proposal on a comprehensive consultative process that was initiated in 2003 for the development of its Poverty Reduction Strategy Paper (PRSP). A central theme of this strategy was modernization of Armenia’s rural economy, especially through infrastructure investment in irrigation and rural roads. The GoA further reviewed and disseminated the PRSP through electronic and printed media and then engaged a broad cross-section of civil society—including rural community members, NGOs, and the private sector—through meetings and roundtables focused specifically on development of a proposal for the compact.

In order to provide an internet-based resource and information portal, the GoA maintained an interactive website that provided program information, meeting minutes, a forum page on which to debate issues related to proposal development and implementation, and an e-mail address for sending inquiries and concerns. As a result of the consultative process, there was broad agreement that identified rural roads and irrigation as fundamental strategic investments to help reduce rural poverty and improve economic growth.

The selection of the irrigation and rural roads projects introduced several challenges, with issues ranging from communal land rights and responsibilities, sustainability of the investments, and civil society involvement in compact development and implementation. To address concerns of the NGO community, which contributed substantially to the proposal development and due diligence processes, the GoA offered to have them elect their own representatives to participate in inter-governmental board of trustees meetings. Issues raised continued to be addressed through a consultative process that incorporated feedback mechanisms, particularly with stakeholders involved in irrigation, agriculture, rural road, policy development and advocacy, and groups that specialized in monitoring and evaluation.

Compact Summary

In March 2006, MCC and the GoA signed a five-year, $235.65 million compact designed to increase the country’s economic growth and reduce poverty by investing in the economic performance of the agricultural sector. The compact sought to advance this goal through strategic investments in rural roads, irrigation infrastructure, technical assistance, and financial support to water supply entities, farmers, and commercial agribusinesses.

The compact consisted of two projects:

• The Irrigated Agriculture Project ($145.67 million) aimed to address the physical, managerial and financial investments needed to generate sustainable increases in rural income through irrigated agriculture. This project consisted of two complementary activities: (i) the Infrastructure Activity and (ii) the Water-to-Market Activity (WtM).
• The Rural Road Rehabilitation Project (RRRP) ($67.1 million) aimed to rehabilitate up to 943 km of the high-priority lifeline network, a network of secondary and local roads which aimed to ensure that all communities, towns and villages were linked to the main road network, either directly or through other communities.

The compact had activities in every region in Armenia, excluding the capital of Yerevan. Through the investment, GoA and MCC expected approximately 426,000 people to benefit over 20 years.

It is important to note that the Armenia Compact, its concepts, project and activity designs, and M&E plans were developed prior to the full operationalization of MCC’s gender and social inclusion (GSI) guidance and evidence-based approaches to compact development, as well as MCC’s investment criteria on Women’s Economic Empowerment. It is possible that MCC’s subsequent GSI and M&E processes would have resulted in a different design, especially with regards to delivering benefits to women and disadvantaged groups, as well as more robust tracking and analysis of gendered and inclusion-oriented disaggregated data, indicators, and outcomes. Consequently, MCC is unable to provide an analysis or more fulsome discussion of the distributional impact of the investments and differentiated socio-economic outcomes of this compact. Read more about MCC’s current approach to gender and inclusion.

The compact was likewise completed prior to MCC’s adoption of the IFC Performance Standards and the MCC Road Safety Policy. Nonetheless, the work in Armenia set some important precedents for MCC’s management of environment and social issues and led to improved practices and approaches in areas such as resettlement, contractor health and safety, revitalizing a RAMSAR 2 site, management of asbestos containing materials, and public consultations. Read more about MCC’s current approach to environmental and social performance.

Under the MCC country ownership model, MCC’s country counterparts are responsible for implementing MCC-funded programs. Partner governments establish accountable entities typically known as Millennium Challenge Accounts to manage implementation for compact projects. In Armenia, MCA-Armenia (MCA-A) was created soon after signing the compact to implement the country’s programs.

By the end of the compact in September 2011, 99.7 percent of the compact budget of $177 million that remained after MCC placed a hold on $58.65 million in RRP funding was spent to reduce rural poverty through a sustainable increase in the economic performance of the agricultural sector in Armenia. Following the conclusion of the compact, the GoA decided that the Foreign Financing Projects Management Center Program Implementation Unit (FFPMC), on behalf of the Ministry of Finance, would be the successor to MCA-A, and would be responsible for post-compact oversight of compact activities to ensure the sustainability of MCC’s investments. A special project management group was formed within the FFPMC, comprised of a small team of full- and part-time previous MCA-A staff, funded by the GoA from the interest revenue generated by the WtM Credit component. This Project Management Group operated for several years, winding down operations by the time the final MCC evaluation was published in 2014.

Irrigated Agriculture Project

Estimated benefits at compact closure correspond to $153.7 million of project funds, where cost-benefit analysis was conducted:

MCC computed economic rates of return (ERR) for both the Infrastructure Activity and the Water-to-Market Activity as summarized in the table below.

Estimated Benefits for Activities

Activity	Time	Estimated Economic Rate of Return (ERR) over 20 years	Estimated beneficiaries over 20 years	Estimated net benefits over 20 years (NPV discounted at 10% per annum in million USD)
Infrastructure Activity	At the time of signing	27.5%	126,000	167.1
	At compact closure 4	24.4%	421,407	145.5
Water-to-Market Activity	At the time of signing	15.5%	124,000	19.1
	At compact closure 5	11.5%	28,831	2.7

Economic analysis of the Irrigated Agriculture Project estimated the benefits attributable to each investment component. In particular, the analysis quantified expected incremental increases in income from newly irrigated land, the increase in high value-added crop cultivation, higher yields, lower production costs, and energy and water savings.

Project Summary

The Irrigated Agriculture Project aimed to increase the productivity of the agricultural sector by extending and improving the quality of the irrigation system, strengthening the entities that managed the system and enabling farmers to commercialize their products. This project included two activities: the Infrastructure Activity and the Water-to-Market (WtM) Activity.

Infrastructure Activity

The Infrastructure Activity (original budget: $113.2 million; total disbursed: $121.5 million) sought to improve dilapidated irrigation infrastructure in order to expand the agricultural land area under irrigated production and improve the overall efficiency of sourcing and delivering water to farmers. The activity addressed these issues by rehabilitating major sections of the country’s main canal systems, modernizing some of the most urgently needed pumping stations, introducing new gravity irrigation schemes, rebuilding tertiary canals, and restoring sections of the drainage system in the Ararat Valley.

During the initial project development stage, MCA-A hired the Water Sector Development and Institutional Strengthening Project Implementation Unit (Irrigation PIU under the State Water Committee of Armenia) to implement the Infrastructure Activity. However, progress was slow, and in 2008, MCA-A terminated the contract and took back responsibilities for implementation.
In 2008, the Infrastructure Activity was re-scoped due to the devaluation of the US dollar against the Armenian dram and construction price escalation. As a result, MCC and the GoA decided to develop a new program ensuring the most optimal mix of components to reduce poverty in the rural areas. The re-scoped activity components included rehabilitation of: (i) main canals, (ii) gravity systems, (iii) tertiary canals, (iv) drainage systems, and (v) pumping stations.

The main canal works included the rehabilitation of six canals to reduce water losses in the main canals, increase irrigated lands, and improve water distribution in the irrigation system. Over 40 kms of new canal lining were refurbished and over 260 water structures were installed. By the end of the compact, $22.7 million was invested in the canals.

The gravity systems 6 works included construction of five gravity systems to help save electricity and provide stable water supply to the communities. Construction of the gravity-fed irrigation systems included almost 50 kms of canal. By the end of the compact, around $8.8 million was invested in the rehabilitation of five gravity systems.

Tertiary canals were rehabilitated in more than 100 communities all over Armenia. These communities were selected from a list of priority communities identified by the GoA and became eligible under a 15 percent co-financing contribution towards the overall construction costs.  Approximately 220 kms of tertiary canals were rehabilitated. By the end of the compact, $15.8 million was invested in tertiary canals.

The rehabilitation of drainage systems reduced ground water levels in the Ararat and Armavir regions, increased crop productivity and arable land in the drainage area, and helped to regulate groundwater levels. This rehabilitation included cleaning 470 kms of canals. By the end of the compact, over $23.1 million was invested in the rehabilitation of drainage systems in the Ararat and Armavir regions.

Rehabilitation of pumping stations in six regions in Armenia ensured more reliable water supply and increased irrigated lands. By the end of the compact, $40.6 million was invested in the rehabilitation of the pumping stations.

Water-to-Market Activity

The Water-to-Market Activity (WtM) (original budget: $32.4 million; total disbursed: $32.2 million) was designed to help farmers use irrigation improvements to introduce new technologies and shift to high-value agriculture (HVA), build the management capacities of the Water Supply Agencies (WSAs) and Water User Associations (WUAs), and provide training and access to credit for member farmers to transition to more profitable, market-oriented agriculture which would improve their income and lead to future economic growth in rural areas. To tackle these issues, the activity was separated into two sub-activities: (a) Improving Profitability of WUA Member Farmers; and (b) Strengthening Irrigation System Entities. These sub-activities are described further below.

During the program implementation, the WtM Activity also underwent other modifications based on recommendations made through different programmatic studies. These changes resulted in improved synchronization of the training component, established linkages between producers and agribusinesses, and increased efficiency of project management.

Improving Profitability of WUA Member Farmers Sub-Activity

The objective of the WtM’s Improving Profitability of WUA Member Farmers Sub-Activity was to ensure that the Irrigation Infrastructure Activity was sustained through a combination of farmer training, access to credit, and support to small agribusinesses. The sub-activity included three components: (1) Farmer Training, (2) Credit, (3) and Post-Harvest, Processing and Marketing.

The Farmer Training component was the strategic tool of the WtM Activity. Overall, two types of trainings were implemented:

• On-Farm Water Management trainings focused on introducing farmers to the newest technologies for on-farm water management in order to secure water savings.
• High-Value Agriculture (HVA) trainings introduced farmers to the new technologies, approaches, and practices for transition to higher-value agriculture.

It was assumed that the adoption of both practices would increase farming income. Farmers were first required to take the on-farm water management training and then the HVA training. The training component was scheduled in a way that a program farmer moved from one training to the next within a relatively short period of time to maintain high levels of interest and ensure a higher degree of adoption. Demonstration sites were directly linked to classroom training components as an extension education program (visits to demonstration sites versus viewing of demonstration videos) and a prerequisite to receiving a training certificate. Combined training demonstration sites were established to ensure the effectiveness of investments and higher income. Directional signage was introduced to promote the usage of demonstration sites and to increase accessibility for villagers.

The Credit component was to support the increase of affordable, longer-term credit to the WtM trainees. The main participants were the farmers who passed the on-farm water management and HVA trainings and wanted to make investments into new technologies. The total amount of credit provided to the banks and credit organizations for five years was USD $8.5 million, with a maturity period of up to seven years and with an annual interest rate of a maximum of 12 percent.

More than 79 percent of compact-funded loans were lent by Universal Credit Organizations, small lenders that are important players in the rural areas and compete with Armenia banks. These loans went to end-borrowers for any productive agricultural purpose associated with HVA crops and technologies promoted by the WtM Activity or directly related agribusiness activity in all regions of Armenia, excluding Yerevan. This component was monitored and implemented by the Rural Finance Facility Project PIU. Eligibility for credit was linked to the training activities, since insufficient access to finance proved to be one of the impediments for farmer adoption of new techniques introduced during the trainings.

Over the course of the compact, credit objectives became more explicit and targeted towards the following agricultural activities: adoption of on-farm water management and new irrigation technologies; establishment of orchards, vineyards, greenhouses, and processing/sale centers; establishment of collection and consolidation centers; and adoption of food safety and quality management systems. By the end of the compact, repaid loan funds were lent again resulting in a total of $12.7 million ($3 billion AMD equivalent) of credit distributed to over 1,000 farmers through existing Armenian banks and other financial institutions. Given its success, after the compact the GoA continued the administration of the credit program (expected through at least  September 2030), using the repaid funds from the initial set of loans.

In November 2019, the extended program was transferred from  the Ministry of Finance’s Foreign Financing Projects Management Center to the Ministry of Economy’s Department of Implementation of Agricultural Programs. The Ministry of Economy reports that as of March 2020, a total of 4,444 loans in the amount of 17.3 billion AMD have been provided to finance the below investments:

Activity Type/Sector	%
Livestock	27.5
Orchards and vineyards	10.1
Food processing	1.6
Greenhouses and cooling facilities	35.6
Leasing	5.7
Vegetable growing and potato cultivation	1.8
Other purposes	17.6
Total	100.0

The Post-Harvest, Processing and Marketing component established linkages between the producers and agribusinesses. To ensure successful operations and to strengthen the capacity of the processing enterprises, this component provided assistance to increase the quality of supplied products; facilitated the establishment of collection points and consolidation centers to store, sort and package produce; and linked farmers to consolidators.

Through these activities, food processors, wholesalers, and institutional buyers were linked to agricultural associations and communities. Project-supported consolidation centers and collection points started their operations in late 2009. More than 15 were established and became operational in 2010 to provide farmers with increased opportunities to sell their produce. By the end of the compact in September 2011, 24 collection centers and three consolidation centers were established, which provided farmers with appropriate conditions to store, sort, and sell their produce close to their land plots, without the transportation cost required to get to markets. Supermarkets (e.g., STAR supermarket) were linked to the Federation of Agricultural Associations, other grower organizations, and producers. A market information system, ARMIS, was developed to provide real-time price and marketing information to all interested parties—from farmers to processors and supermarkets. Exporters were supported under a unified “Fruits from Armenia” brand and a trial shipment of modern packed and palletized produce was initiated to Russian supermarkets. In 2009, MCA-A initiated two big public events—Buy Armenian Campaign and WtM-EXPO—to increase interest toward Armenian-made products. In December 2009, the GoA also adopted legislation on agricultural cooperatives, thereby meeting one of the compact’s conditions precedent.

Strengthening Irrigation System Entities Sub-Activity

Through the WtM’s Strengthening Irrigation System Entities Sub-Activity, MCA-A supported management reforms in the irrigation sector by developing and strengthening the capacity and effectiveness of irrigation system management entities, including WUAs and WSAs. The sub-activity sought to enhance the institutional capacity of 44 WUAs and three WSAs to maintain the rehabilitated infrastructure. MCA-A strengthened these institutions by helping them develop and implement Management Improvement Plans. In addition, WUAs received improved water management tools, including 180 computers with sophisticated budgeting, accounting and georgraphic information system software; 44 units of welding equipment; and 38 backhoe loaders to increase the efficiency of their maintenance operations. It was expected that as a result, WUAs could better monitor and manage water supply and demand, improve collection of water fees and membership dues, and enhance water user awareness of and involvement in the WUA management.

At completion, the Irrigated Agriculture Project represented one of the largest investments to date into the rehabilitation of Armenia’s vital irrigation infrastructure as well as provided extensive technical and financial assistance to rural farmers and to the government irrigation entities that support them.

However, during implementation, the various project activities were disjointed in targeting project beneficiaries and sequencing. The irrigation infrastructure rehabilitation activities were significantly reduced in scope and faced construction delays of two to three years. Despite irrigation construction scope changes and delays, farmer training in improved technologies commenced for farmers that already had some access to reliable water and/or would benefit from irrigation rehabilitation efforts. The estimates of reliable access to water for farmers trained ahead of the irrigation rehabilitation proved to be inaccurate for some communities. Credit on-lending and training of financial institutions began approximately at the same time as farmer training efforts. Post-harvest, processing, and marketing activities scaled up approximately two years after farmer training began and completed major initiatives, such as the creation of consolidation centers, in the final year of implementation. Finally, water user association capacity building occurred alongside other project activities with strong complementarities with the on-farm water management training content, although actual access to water varied. Infrastructure rehabilitation also was completed only in the final year of the compact.

Key performance indicators and outputs at compact end date

Indicators	Baseline (2006)	Actual Achieved (2011)	End of Compact Target 7 (2011)	Percent Complete 8
Primary canals rehabilitated	0	41.8 km	41.8 km	100%
Pumping stations renovated	0	17	17	100%
Conversion from pumping to gravity completed	0	5	4	125%
Tertiary canals rehabilitated	0	232.8 km	220 km	106%
Drainage canals cleaned	0	470 km	470 km	100%
Government budgetary allocations for maintenance of irrigation system	Unavailable	1,385,538 Armenian drams	1,500,000 Armenian drams	92%
Training/technical assistance provided for on-farm water mgmt (Total)	0	45,639 participants	45,000 participants	101%
Training/technical assistance provided for on-farm water mgmt – Female	0	14,520 women	No Target	No Target
Training/technical assistance provided for HVA (Total)	0	36,070 participants	36,000 participants	100%
Training/technical assistance provided for HVA – Female	0	12,189 women	No Target	No Target
Loans provided	0	$12,700,000	$8,500,000	150%
Loan borrowers (Total)	0	1,008	No Target	No Target
Loan borrowers – Female	0	106	No Target	No Target
Training/technical assistance provided for post-harvest, processing, and marketing	0	227 businesses	225 businesses	101%
Recovery of WUA operations and maintenance cost by water charges	36.7%	48.1%	60%	49%
Increased collection of irrigation service fee for the water used	50%	68.7%	55%	374%
Management Improvement Plans developed for WUAs	0	44	44	100%

Explanation of Results:

WUAs did not meet the targeted operations and maintenance cost recovery rates during implementation or after. Improvements were made, but the cost recovery rates leveled off after 2010. Cost recovery rates were not expected to improve enough to meet the target without increasing water charges.

Water user associations increased their irrigation service fee collection rates. As noted in the WtM evaluation, further discussed below, the sharp increases in service fee collection in 2009 and 2010 may have been linked to the GoA’s decision to provide free irrigation water in April and May of 2009 and 2010 in an effort to alleviate agricultural hardship during those years. Because water users’ service fee obligations were substantially lower during these years, the overall service fee collection rate increased despite an actual decrease in total revenues from water payments during this time period. However, the increase in service fee rates achieved in 2010 were sustained from 2011-2013. This suggests that water user associations were able to increase irrigation service fee rates materially with improved administration, and the increases were not solely due to the GoA’s decision to provide free irrigation water in some months. 9

Evaluation Findings

The Irrigated Agriculture Project was evaluated through multiple evaluations. An initial evaluation report was produced in 2013 to report on results from the WtM Activity. Then a second evaluation report was produced in 2016 covering the Infrastructure Activity and additional analysis on the related training and institutional strengthening interventions from the WtM Activity.

Water-to-Market Activity

Status of the Evaluation

Component	Status
Baseline Report	Completed in 2008. Report is public.
Endline Report	Completed in 2013. Report and de-identified data are public.

Irrigation Infrastructure Activity

Status of the Evaluation

Component	Status
Baseline Report	Completed in 2010. Report is public.
Endline Report	Completed in 2016. Report and de-identified data are public.

The program logic for the Irrigated Agriculture Project assumed that when farmers had reliable access to irrigated water; were trained on water management and productivity techniques; had access to credit, stronger post-harvest and marketing channels, improved irrigation delivery, and administrative management services provided by water user associations; they would apply techniques and investment that transitioned their farm operations from low-value to high-value agricultural production. This was expected to result in additional farm income from increased agricultural productivity and market access, which was assumed to lead to an increase in overall household income.

For each of the four components of the WtM Activity, the WtM Activity evaluation examined the following two broad sets of questions:

1. How was the component implemented? What were the characteristics of each component’s participants, and how were these participants identified and recruited? What assistance was provided to participants through the component?
2. What were the impacts of the component? What were the impacts on practices or use of new technologies as a result of the component? What were the impacts on key outcomes such as household income and poverty?

The Irrigation Infrastructure Activity evaluation examined the following research questions:

1. Did the program affect the quantity and reliability of irrigation water provided to Armenian farmers?
2. Did farmers adopt new agricultural practices as a result of the program? Further, is there evidence that farmers who received Water-to-Market training postponed adopting new agricultural practices until after the irrigation infrastructure had been rehabilitated?
3. Did the program affect agricultural productivity?
4. Did the program improve household well-being for farmers served by the rehabilitated infrastructure, especially income and poverty?
5. Is there evidence that the infrastructure investments will be sustained after rehabilitation was complete? Are the water user associations themselves financially sustainable?
6. Were the program effects large enough to justify its costs?

The WtM evaluation report found that the Farmer Training component did not result in an increase of on-farm water management practices, transition to high-value crops, productive income, or household income. Some positive impacts were detected on simple high-value practices such as soil preparation and purchase of pesticides from a licensed store. The Infrastructure Activity was significantly delayed during the compact period, and the farmer training implementation and initial evaluation continued without the new and improved infrastructure (a projected amount of 9,000 new hectares under irrigation and 38,000 hectares of improved irrigation). This was a fundamental breakdown in the program logic and a key lesson learned for MCC.

For the Credit component, the independent performance evaluation detected potential effects on adoption, crop production, and household income. WtM credit recipients were more likely to make agricultural investments, had higher production, and realized larger incomes. However, the evaluation has some important limitations (like not accounting for unobserved differences between treatment and comparison groups, such as motivation), which means that the results are likely upwardly biased.

For the Post-Harvest, Processing, and Marketing component, the independent performance evaluation found that the majority of recipients of this assistance reported positive outcomes, including improved product and service quality, increased productivity, and increased sales. However, the evaluation has some important limitations: all results are ex-post results reported by assisted enterprises and the results do not take into account what would have happened without the assistance, nor do they take into account any baseline information.

For the Strengthening Irrigation System Entities component, the independent performance evaluation reports that membership rates and membership fee payment rates increased moderately during implementation. In addition, WUAs improved their financial standing, but they were not yet approaching financial self-sufficiency. These improvements cannot be attributed fully to the component, because many other factors affect these outcomes, such as weather and national policies. For additional information on the WtM evaluation, see the Evaluation Brief.

The Infrastructure Activity evaluation found that while farmers near tertiary canals perceived improvements in the timeliness and reliability of irrigation water, others near rehabilitated large infrastructure did not. There was also no evidence that farmers irrigated more land, that production of HVA crops or their yields increased, or that household income and consumption increased. For additional information on the Irrigation Activity evaluation, see the Evaluation Brief.

The results from the WtM Activity training evaluation were disappointing; however, data was collected in 2013 before all of the irrigation system improvements were completed. Therefore, the 2016 Infrastructure Activity evaluation revisited the training and institutional strengthening components. For communities that received WtM training in 2008, the evaluation did not find that adoption of on-farm water management practices was substantially higher for farmers in 2013 than it was in 2010. The levels of adoption of profitable new practices remained low. In addition, WUAs’ financial status had leveled off since 2010, with little additional progress toward financial self-sustainability.

Evaluation Learning 10 :

Lessons learned from the WtM Activity evaluation included:

• Always return to the program logic. It is especially important in integrated projects that the rollout is coordinated with complementary activities.
• Balance ambitious targets with training effectiveness.
• The randomized roll-out evaluation approach has risks.
• A multifaceted development approach requires proactive and visionary management.

Lessons learned from the Irrigation Infrastructure Activity evaluation included:

• Root cause analysis is critical to successful project design, especially where behavior change is integral to the program logic. 
• Supporting the development of new institutions, such as WUAs, is inherently difficult and can be a long-term undertaking, so MCC projects should account for the five-year timeline and anticipate necessary follow-on activities.
• During a project re-scoping, the program logic, economic analysis, potential beneficiaries, and evaluation plan should be re-assessed in a cohesive way by a coordinated project team.
• Better water monitoring tools could help WUAs while also providing better measurement of outcomes.
• An irrigation project where demand for the intervention exceeds available funding can provide an opportunity for random assignment (or other allocation mechanisms that are fair, efficient and informative) that can improve the level of rigor of evaluation results.
• Sector-specific technical capabilities should be required on the evaluation team when needed to assess key intermediate outcomes.
• Evaluation questions are based on the program logic and must be designed carefully from the beginning to understand the scope and limitations of the evaluation.

The Rural Roads Rehabilitation Project

Estimated benefits at compact closure correspond to $8.4 million of project funds, where cost-benefit analysis was conducted

Project Economic Analysis

MCC computed economic rates of return for the Rural Roads Rehabilitation Project as summarized in the table below.

Activity	Time	Estimated Economic Rate of Return (ERR) over 20 years	Estimated beneficiaries over 20 years	Estimated net benefits over 20 years (NPV discounted at 10% per annum in million USD)
Rural Roads Rehabilitation Project	At the time of signing	25.9%	142,905	80.52
	At compact closure	17.6%	6,356	2.24

The benefits of the Rural Road Rehabilitation Project (RRRP) are attributable to (1) operating cost savings for vehicles and wear and tear on the roadbed, and (2) travel time-savings, as improvements in road condition permit higher average driving speeds. These reductions in the cost and time of transport translate to enhanced economic growth via reductions in the cost of production and distribution and by making other factors of production more productive for enterprises and individual farmers alike.

Project Summary

The objective of the Rural Roads Rehabilitation Project (RRRP) (original budget: $67.1 million; total disbursed: $8.4 million) was to expand communities’ access to agricultural markets, non-farm income opportunities, and social infrastructure by improving the condition of rural roads. To accomplish this, the project was designed to rehabilitate about one-third of the high priority rural roads in the lifeline network, including 943 kilometers of rural roads in 85 road segments throughout the country. This included 321 km of secondary roads and 622 km of local roads, for a total cost of USD $67.1 million. All rehabilitation works were to be on existing infrastructure and would include pavement rehabilitation, improvements on up to 19 bridges, drainage facilities, and road safety features.

Initially, it was estimated that approximately 360,000 rural inhabitants in 260 rural communities would benefit. However, during compact implementation, the RRRP underwent significant changes, primarily due to the devaluation of the US dollar against the Armenian dram and construction price escalation.

In 2007, MCA-A initiated the fast-track construction of roads and irrigation infrastructure (tertiaries) to show some work in progress. For those pilot constructions, MCA-A used ready-to-go designs prepared by other donors since its own feasibility and design activities were still underway. However, during the feasibility and design stage later that year, it was determined that MCA-A would not be able to fund all the roads that were originally included in the program.

To determine next steps, MCA-A worked with the Armenian Roads Directorate, the RRRP implementing entity responsible for technical oversight of project implementation, to restructure the project. The revised packaging was based on new ERR calculations derived from feasibility studies and designs submitted by a design consultant. As a result, in November 2007, the project scope was reduced by roughly two-thirds. Around 300 kilometers of rural roads, all of which had ERRs of at least 12.5 percent, were selected for rehabilitation, with the same total cost of USD $67.1 million. Overall, 84 communities located in different regions of Armenia were slated to benefit.

Accordingly, by June 2009 only the pilot H17 Armavir – Isahakyan – Gyumri road section (24.4 km) was fully rehabilitated using MCC funding (this pilot had been underway and completed prior to the events in March 2008). This road section was constructed under the MCA-A fast track initiative. It connects a number of villages in the Armavir region with Gyumri, the country’s second biggest city. The rehabilitated road serves 12 rural communities and reaches 6,356 beneficiaries.

Following the funding hold in May 2008, the GoA decided to fund the project and allocated USD $16,800,000 to rehabilitate initially planned road sections. By June 2009, some works, such as earthworks, pothole patching, asphalt concrete layering, and rehabilitation of sidewalks, were already implemented under the re-scoped RRRP through the GoA funding. Prior to rehabilitation works, the MCA-A roads team, together with Armenian Roads Directorate and the feasibility and design consultant, had conducted a series of community outreach meetings to inform the local authorities and actual beneficiaries—farmers—about the future construction works.

As a result of the hold and the end of MCC funding for roads construction and rehabilitation, on September 25, 2009, MCA-A proceeded to transfer the RRRP to the Ministry of Transport and Communication of Armenia. Also, MCA-A, the contractor, and the Ministry of Transport and Communication signed agreements to amend and transfer the two contracts for the construction of the Armavir-Isahakyan-Gyumri road sections. On the same date, the assets and documents related to RRRP were transferred to the Ministry of Transport and Communication. MCA-A’s contract with the Armenian Road Directorate, the RRRP implementing entity,  was terminated on September 30, 2009.

Even though this project was cut short due to the above mentioned circumstances, it did accomplish several achievements, including the rehabilitation of 24.4km of rural roads; development of designs for 575 km of rural roads; increased knowledge of best maintenance practices through study tours to Sweden, Finland, and Estonia; and production of guidelines for winter maintenance contracts.

The impact of this project was further extended by the fact that the World Bank financed the construction or rehabilitation of approximately 150 km of rural roads using the MCA-A funded detailed designs.

Key performance indicators and outputs at compact end date

Indicators	Baseline (2006)	Actual Achieved (2008)	End of Compact Target (2011)	Percent Complete 11
Government budgetary allocations for rehabilitation of road sections in the road lifeline network	Not available	7,227,000 Armenian drams	3,310,000 Armenian drams	218%
Government budgetary allocations for routine maintenance of the entire road network	Not available	6,656,000 Armenian drams	6,290,000 Armenian drams	106%
Average daily traffic on pilot roads	637	735	706	142%
International Roughness Index for pilot roads 12 (meters per kilometer)	14.16	3.47	4	105%
Pilot road sections rehabilitated (kilometers)	0	24.4	24.4	100%

Explanation of Results:

The average daily traffic on pilot roads increased; however, it can not necessarily be attributed to the road rehabilitation project without having an estimate of how much traffic would have changed in the absence of the rehabilitation. Traffic could have increased (or decreased) due to other factors outside of the RRRP. 13

Evaluation Findings

Even though funding was put on hold, eventually indefinitely, for the RRRP, MCC continued with the independent impact evaluation, which covered roads designed using MCC funds but built as part of the World Bank project. During the compact, MCC funded an increase in the sample size of the Integrated Survey of Living Standards (ISLS), the Armenia national household survey, conducted by the National Statistical Service of Armenia on an annual basis. MCC’s independent evaluators planned on using the ISLS to evaluate the impact of RRRP before the project was placed on indefinite hold. Since the increase in sample of the ISLS was not tied to the RRRP, the data collection continued until the end of the compact and was provided to MCC. Both the GoA and the World Bank expressed interest in having MCC’s independent evaluator analyze the ISLS data as previously planned to provide insights into the impacts of rural road rehabilitation. In addition, MCC saw value in completing the evaluation because of the large percentage of MCC’s portfolio invested in roads.

Status of the Evaluation

Component	Status
Endline Report	Completed in 2015. Report and de-identified data are public.

The original RRRP was designed to improve the quality of the lifeline road network in Armenia in order to enhance the economic performance of the agricultural sector. It was expected that improved road quality would reduce transportation costs and increase vehicular activity, which would increase access to markets and social infrastructure. As a result, farmers would be able to access agricultural inputs at cheaper prices and an increased number of retailers and buyers of agricultural products could access the communities, thus creating conditions for farmers to sell their agricultural production at a higher price. These changes were expected to incentivize farmers to invest more, thereby increasing employment and production, leading to improved performance of the agricultural sector and poverty reduction.

The evaluation included the following research questions:

1. Did rehabilitating roads affect the quality of roads?
2. id rehabilitating roads improve access to markets and social infrastructure?
3. Did rehabilitating roads improve income from employment?
4. Did rehabilitating roads affect agricultural productivity and profits, and if so, by how much?
5. Did rehabilitating roads improve household well-being for communities served by these roads, especially income and poverty?

The independent impact evaluation found that road rehabilitation efforts can improve road quality and increase the use of roads in the short term. There was a 39 percentage point increase in favorability rating of regional roads and a 20 percentage point decrease in market access difficulties. However, those changes were not sufficient to stimulate increases in agricultural production and sales in a timeframe of one to two years. For additional information on the road evaluation, see the Evaluation Brief.

Evaluation Learning 14 :

Several key lessons learned from the Rural Roads Rehabilitation Project evaluation contributed to a broad set of lessons derived from other roads evaluations and the findings of MCC’s Transport Sector Practice Group’s internal reviews. These lessons included:

• Base evaluation decisions on a clear program logic.
• Set realistic time horizons and keep data collection plans flexible.
• Ensure sufficient statistical power.
• Conduct road network analysis.
• Consider alternatives.
• Address road maintenance constraints.
• Improve road investment economic assessments.
• Ensure collection of high-quality data.
• Enforce design review standards.

Compact Changes

The Infrastructure Activity was re-scoped in April 2008 due to the devaluation of the US dollar against the Armenian dram and construction price escalation. As a result, a decision was made to develop a new program to ensure an optimal mix of components to reduce poverty in rural areas. Consequently, the number of gravity systems dropped from 18 to 5; reservoirs from 7 to 0; and pumping stations from 68 to 17. The number of additional hectares under irrigation was revised from 20,340 to 1,767. Also, because of certain linkages and connections between the Infrastructure Activity and the WtM Activity, after the re-scope, the WtM targets and outcomes were reviewed and modified in order to reach greater impact with smaller outputs. The request for modification came from the WtM implementer and the output targets for on-farm water management and HVA components were eventually revised in 2009 based on the revised ERR analysis. In addition, it became apparent that there were significantly less registered processing enterprises in Armenia than the defined target. Consequently, all WtM targets included in the MCA-A Monitoring & Evaluation Plan were revised to reflect programmatic changes and the ERRs were recalculated. The target for the number of farmers trained in on-farm water management was reduced from 60,000 to 45,000 while the number trained in HVA was increased from 30,000 to 36,000.

Also, in May 2008, MCC placed a hold on the disbursement of additional funding for the RRRP (approximately $58.65 million) due to post-election events in March 2008 that MCC determined to be inconsistent with MCC’s eligibility criteria, particularly with regard to democratic governance. The hold was endorsed by MCC’s Board at its June 2009 meeting, after which the Board announced MCC would not resume any further funding for the roads project for the duration of the compact.

The original number of kilometers of road to be improved under the compact was 943 km. After the re-scoping in April 2008, the target was reduced to 332 km. However, as a result of the hold on funding, only 3 percent of the original target was completed.

Coordination and Partnerships

MCC and MCA-A, the World Bank, and other donors, including the Lincy Foundation, the US Department of Agriculture, Oxfam, and USAID, coordinated investments in Armenian infrastructure development, building capacity of rural farmers and businesses, data collection for monitoring and evaluation, and strengthening the institutional framework in the agricultural sector throughout compact implementation.

In 2007, MCA-A used tertiary canal designs prepared through a World Bank tender to accelerate construction repairs to tertiary canals in four communities ahead of schedule. Also in 2007, MCC used road designs prepared with funding from the Lincy Foundation to carry out repairs on a 24.4 km section of rural road in advance of signing major works contracts for road repairs.

Following the decision to put a hold on funding for the Armenia roads project in May 2008, 150 km out of 551 km of MCC-funded road designs were shared with, tendered out, and repaired by the World Bank. The GoA also invested $16.8 million to advance the construction of roads sections that were to be funded by the compact.

Conditions Precedent

Key Policy Conditions Precedent
(CP)

Key Compact Component(s)	Major CP or Policy Reform Required	Rating Met on Time/Deferred/Waived
Compact wide	The Central Banking Authority will modify the regulatory framework for the licensing of MFIs as credit organizations.	Met
Irrigated Infrastructure Project – Infrastructure Activity	MCA-A has arranged for 15% WUA co-financing of investment in tertiary canal systems consistent with the World Bank Irrigation Development Project.	Met
Irrigated Infrastructure Project – Infrastructure Activity	The GoA has developed and adopted legislation and/or regulations, or amended existing legislation and regulation, to facilitate the establishment of agricultural cooperatives.	Met
Irrigated Infrastructure Project – Infrastructure Activity	The GoA has developed and adopted legislation and/or regulations to improve statistics on agriculture, including an accounting of the sector and data collection.	Met
Irrigated Infrastructure Project – Infrastructure Activity	The GoA adopted the amendment of Article 16 of the Law on Water Users Associations and Federations of Water Users Associations proposed by the Government.	Met