The Office of Development Effectiveness
ODE Work Plan
The ODE Work Plan below has been endorsed by the Independent Evaluation Committee and approved by the DFAT Secretary.
The Office of Development Effectiveness (ODE) is a branch within the Department of Foreign Affairs and Trade. ODE monitors the Australian aid program's performance, evaluates its impact, and contributes to international evidence and debate about aid and development effectiveness.
This work plan has been reviewed and endorsed by the Independent Evaluation Committee (IEC), an external advisory body that oversees all of ODE's work. The role of the IEC is to strengthen the quality, credibility, influence and independence of ODE's work through the provision of independent strategic and technical advice.
This work plan includes all work scheduled to be completed in 2019, and evaluations which ODE expects to commence in 2019 and complete in 2020. The plan is approved by the DFAT Secretary.
2. Overview of ODE Work Plan
ODE will complete four strategic evaluations in both 2019 and 2020. This includes developing an approach to undertaking evaluations in 2020 and beyond that would examine DFAT's development engagement across aid, trade, security and diplomacy with a single country.
A measured level of strategic evaluation work will enable ODE staff to undertake more engagement with program areas. This includes substantial involvement in a program evaluation, an increased role in quality assuring program evaluations for the department's high-value investments, and increased engagement with Divisions to support implementation of the DFAT annual aid evaluation plan. In addition, ODE will independently quality assure all Final Aid Quality Checks (FAQCs). This is a fourfold increase in scope from the trials of 2017 and 2018, which covered a sample of 20 FAQCs.
ODE will continue to build evaluation capability across the department, and seek to provide 'just in time' evidence to support decision making.
The work plan is organised under five headings: strategic evaluations, performance and quality analysis, evaluation policy implementation, building capability, and knowledge brokering. Details of the work in each of these areas is elaborated below.
2.1 Strategic evaluations
Strategic evaluations are assessments of Australian aid that focus on key policy directions, specific development themes and sectors, or large programs. These evaluations are initiated, managed and conducted by ODE. They typically examine a number of investments, often across multiple countries, regions or sectors. By virtue of their scope, strategic evaluations tend to be time and resource intensive.
Strategic evaluations provide independent oversight of Australian aid. They inform change in both strategy setting and operational practice and they assist in identifying new and innovative ways to deliver aid.
The following principles guide the selection of evaluation topics:
- Policy relevance: assessing progress on implementation of the Australian Government's policy priorities, including the Foreign Policy White Paper.
- Potential learning benefits: focusing on areas where there is a strong demand for insight, information or guidance within DFAT or contributing to global knowledge.
- Coverage and materiality: over time, ODE's evaluations should provide appropriate coverage of expenditure by sector, theme, country and region.
- Risks to aid effectiveness: examination of issues which pose a challenge to the Australian aid program and its reputation.
Table 1: Strategic evaluations
|Topics - to publish in 2019|
|Health support to the Pacific|
|Ending violence against women and girls|
|Global programs - multilateral organisations|
|Technical and vocational education and training|
|Topics - to publish in 2020|
|Facilities for effective aid|
|Conflict and violence|
|Country level evaluation|
2.2 Performance and quality analysis
ODE's performance and quality analysis (PQA) work monitors DFAT's aid performance management and reporting systems on an annual basis, and independently quality assures the assessments they produce. These are undertaken mostly within set timeframes, while some activities are ad hoc and undertaken within a timeframe determined by ODE. There are a number of components to this work, as indicated below.
In 2019 ODE will expand significantly its quality assurance of FAQCs to cover all FAQCs. Recognising its value as a quality assurance process, ODE will consider the possibility of future spot checks of Aid Quality Checks (AQCs). ODE will consult with the Contracting and Aid Management Division and the Independent Evaluation Committee (IEC) in finalising its approach.
Details of all planned work topics are at Attachment 1.
Table 4: Work to complete annually
|Validation of Final Aid Quality Checks|
|Annual quality review of Aid Program Performance Reports|
|Annual quality assurance of Performance of Australian Aid report|
|Review of uptake of ODE recommendations|
2.3 Evaluation policy implementation
Under the evaluation policy ODE compiles an annual aid evaluation plan, which includes program evaluations and ODE's strategic evaluations, for consideration by the Independent Evaluation Committee and for approval by the DFAT Secretary. ODE also conducts analysis on the characteristics of program evaluations with regard to geographic and sector coverage. ODE reports to the Secretary and the IEC on the implementation of the plan mid-year, and on the outcome of the plan early in the following year.
In 2019 ODE has produced the 2019 aid evaluation plan and reported on the outcome of the 2018 aid evaluation plan. ODE will provide greater support to country and regional programs in planning and implementing their evaluations by establishing dedicated geographic or sector contact points within ODE.
From time to time ODE assesses the quality of program evaluations. These reviews (meta-evaluations) are treated as strategic evaluations and listed in section 2.1 of the work plan. An ODE review of 2017 evaluations was completed in 2018. A similar review of 2018 evaluations is not planned for 2019 but we will consider one in future years.
2.4 Building evaluation capability
Managing an evaluation requires significant knowledge, skill and experience. ODE is committed to building evaluation capability and technical skills to support DFAT staff in planning and conducting evaluations, and undertaking performance monitoring.
ODE builds organisational capability through providing guidance and advice to other parts of the Department, assisting with monitoring and evaluation training, and working with program areas on specific program evaluations. From 2019 ODE will have a role in quality assuring evaluations of investments valued at $50 million or greater.
Guidance and advice
ODE maintains guidance material on how to manage an evaluation to support learning on the job. This includes a guidance note available through the Aid Programming Guide and good practice examples of evaluation terms of reference, evaluation plans and management responses on the ODE webpage. ODE also provides ad hoc advice and quality assurance to evaluation managers as requested.
Monitoring and evaluation training
Learning continues for staff in ODE and across the department in evaluation approaches and methods. ODE is made up of a mix of experienced aid program managers and evaluation specialists who are members of the Australasian Evaluation Society through the DFAT corporate membership.
ODE has contributed to the development of content and the delivery of internal training on monitoring and evaluation, and will continue to have an active role in 2019.
Quality assurance of high value investment evaluations
The ODE quality review of 2017 program evaluations found that program evaluations are a credible source of evidence, but that the quality was inadequate for a number of high value investments. This finding suggests a need for additional quality assurance to ensure that high value investments are evaluated effectively. ODE will support evaluation quality by reviewing the terms of reference, evaluation plans and reports for evaluations of investments valued at $50 million or greater.
Program evaluations are independent assessments of the performance of Australian aid investments. They provide evidence to inform program management decisions, contribute to broader learning and help to enhance accountability. Program evaluations are generally commissioned and managed by program areas and may be conducted at whichever point in time is of most value from a management perspective.
Consistent with its role in providing expanded support for program evaluations, and the selection principles outlined above, ODE sometimes funds and leads, or otherwise participates in, a limited number of program evaluations.
ODE's involvement in program evaluations is guided by principles outlined in the introductory section of this work plan in addition to the following considerations:
- Readiness of program areas to collaborate with ODE.
- Clarity of evaluation purpose; be it accountability, learning or both.
- Feasibility of the evaluation timing and the availability of human and financial resources.
- Opportunity to model and promote 'good evaluation practice' within DFAT.
In 2019 an ODE staff member will participate in a program evaluation looking at Australia's humanitarian response in Syria, at the request of the program area.
ODE will continue to expand outreach of its work and the promotion of good evaluation practice domestically and internationally. This includes continuing the practice of co-hosting evaluation forums with ANU's Development Policy Centre and engaging with the Canberra evaluation community of practice and with other APS agencies.
2.5 Knowledge Brokering
ODE plays a role in making information and knowledge available to policy and program managers. DFAT has generated a large body of information through its evaluations, and ODE will support greater use of this information to feed into decisions on future policies and investments.
In addition to overseeing the DFAT aid evaluation policy, which requires evaluations to be published on DFAT's website, ODE increases accessibility by maintaining a centralised list of program level evaluations linked to the ODE website. In 2019, ODE will have a greater focus on producing information products from DFAT evaluations, such as evidence briefs on current issues, to expand the use of information and make relevant learning available to decision makers.
Attachment 1–Description of planned work topics
Table 6: Topics to complete in 2019
|Health support to the Pacific||ODE will examine the quality of DFAT's health programs, whether and how they align with global good practice and whether they are 'fit' for the unique context of the Pacific. The findings will inform bilateral and regional programming in future, improve cross-sectoral collaboration, and guide support provided by Canberra.|
|Ending violence against women and girls||Addressing Violence Against Women and Girls (VAWG) is a priority for Australia, at home and internationally. The evaluation will assess our support to prevent and respond to VAWG in developing countries–including the Pacific where gender-based violence rates are among the world's highest.|
|Global programs||This evaluation is considering how Australia can achieve its Foreign Policy White Paper (FPWP) priorities through multilateral organisations (MLOs). It will draw on case studies across a sample of MLOs, and analyse how Australia has worked with MLOs in the past and make recommendations for how Australia can work with MLOs in the future.|
|Technical and vocational education and training||The evaluation will look at a selection of bilateral and regional skills development investments in the Pacific and identify factors that contributed to success in relation to sustainability and effectiveness. The evaluation will draw heavily on existing program evaluations.|
|Validation of Final Aid Quality Checks||ODE will validate Final Aid Quality Checks (FAQCs). This will provide quality assurance to underpin the Department's reporting against the proposed new target of aid effectiveness and the annual Performance of Australian Aid Report. The method to be used to validate FAQCs will build on the FAQC validation trial undertaken in 2017 and 2018.|
|Annual review of Aid Program Performance Reports||ODE assesses the quality of annual APPRs which report on the performance of country and regional aid programs, highlighting areas of strength and opportunities for improvement. APPR quality reviews have been carried out every year since APPRs were introduced in 2008 and have been published on the ODE website since 2013.|
|Quality assurance of Performance of Australian Aid report 2017-18||As in past years, ODE will independently analyse the performance assessments made in the Performance of Australian Aid report.|
|Review of uptake of ODE recommendations||The management response to ODE's strategic evaluations are cleared at a high level and become a public commitment to implementation of accepted recommendations. ODE reviews the implementation of actions to achieve recommendations around twelve to eighteen months after each evaluation and reports on progress to the IEC and the DFAT Executive.|
Table 8: Strategic Evaluation Topics to complete in 2020
|Pacific Economic Infrastructure||This evaluation will provide evidence for DFAT and its investment partners about how Australia can maximise development outcomes and promote Australia's White Paper strategic and policy priorities through its economic infrastructure investments in the Pacific. The findings and recommendations will inform future DFAT investments and policy engagement on economic infrastructure in the Pacific.|
|Facilities for effective aid delivery||Facilities are broadly defined as flexible, overarching programs implemented through contracts where the outcomes are specified, but the implementation pathways to delivering on those objectives are not specified. This evaluation would examine the effectiveness and efficiency of facilities in delivering outcomes and make recommendations for the future.|
|Conflict and violence||The evaluation will document and assess the relevance and performance of DFAT's policy and interventions in fragile and conflict and violence affected situations; and make recommendations to guide future work.|
|Country level evaluation||The evaluation will examine DFAT's development engagement with a single country across aid, trade, security and diplomacy.|