1.1 Purpose

This evaluation assesses the effectiveness of Australia's response to the food security and refugee crisis in Somalia, Ethiopia and Kenya in 2011, known as the Horn of Africa (HoA) crisis. The purpose of this evaluation is to identify how to improve the effectiveness of Australian humanitarian assistance to future slow-onset crises. The areas of enquiry were:

• how Australian responded
• how Australian assistance was delivered
• how well the needs of affected people were met
• achievements of the Australian response.

1.2 Methodology

Process

The evaluation aimed to be participatory while maintaining rigour and evidence standards. The first step was to develop an evaluation plan that outlined the methods and timeframe. This was circulated to internal stakeholders for comment and was subject to scrutiny by the Independent Evaluation Committee.

Interviews with stakeholders were conducted face to face in Canberra, Melbourne and Nairobi, or by telephone. These interviews were complemented by an intensive process of collecting, sifting and analysing reports, the academic and grey literature, and other relevant material.

The evaluation sought to triangulate findings through a variety of means. This included a review of evaluative material examining the performance of the Australian aid program and implementing partners, as well as general findings relating to areas of the response that Australia funded. The analytical process included an interrogation of financial data with a view to understanding sector and geographic coverage, and for collating and codifying results data. Interviews were written up and analysed for patterns and common themes. An initial feedback session was held in Nairobi and a validation workshop presented initial findings in Canberra to begin the process of developing recommendations.

Evidence

Interviews focused on staff in the Australian aid program, implementing partners and donor organisations (Table 1). The interviews were semi structured, loosely following the four areas of enquiry. They were a strong source of evidence. People were generally candid and frank, and appeared comfortable to share all forms of information verbally. Interviews provided a high degree of consistent information, with major themes emerging quickly. Workshops were held with staff from the Department of Foreign Affairs and Trade (DFAT) to develop recommendations, and focus group discussions were undertaken with beneficiary communities in northern Kenya.

Table 1 Number and type of interviews

AusAID = Australian Agency for International Development; DFAT = Department of Foreign Affairs and Trade; NGO = non-government organisation; UN = United Nations

The secondary data analysis used two main sources of information–agency reporting and external material relating either to the HoA crisis or to the international aid operation. This latter category included evaluations and academic literature (Table 2).

A number of ways were used to review the secondary data. A trawl of the academic literature and general evaluative publications was undertaken to draw out key general lessons appropriate to countries, sectors or agencies, and to look for specific mentions of Australia or AusAID. The documentation provided by the Australian aid program and its partners was reviewed to inform the narrative and factual base for this evaluation, as well as for any pertinent lessons. Reports of, financial allocations and beneficiary numbers were reviewed to compile new analyses of results including where funds were spent and how many people were assisted.

To make evaluative judgements, the extensive humanitarian experience of the team and their knowledge of the HoA crisis were used, along with the evidence gathered in the course of this evaluation. For details of the strength of evidence in each of the areas of enquiry, see Table 3.

Table 2 Type of secondary data review

Table 3 Strength of the evidence

Limitations

The evaluation was mostly based on secondary data. The strength of the findings was dependent on the quality and quantity of data provided, which was less than ideal. There were significant gaps in data and information. Some of this is understandable given the context, because it is difficult to monitor and collect data in humanitarian and conflict situations. Deficiencies in reporting, however, also stem from poor practice in the humanitarian sector, which traditionally has not prioritised accountability (either to donors or beneficiaries) and has not invested in rigorous measurement of effectiveness. Some agencies were reluctant to generate additional information for this evaluation. The United Nations (UN) system proved the most problematic in terms of providing both budgetary analysis and numbers of people reached or assisted. This was primarily because the reporting that Australia requires from these agencies is minimal. Since UN agencies received the bulk of funds, evidence on results is weak.

References

1 London School of Hygiene and Tropical Medicine & John Hopkins University Bloomberg School of Public Health. Mortality among populations of southern and central Somalia affected by severe food insecurity and famine during 2010–2012, for UN Food and Agriculture Organization, Famine Early Warning System Network & Food Security and Nutrition Analysis Unit for Somalia, Rome and Washington, 2013.