Good practice evaluation examples
Good practice evaluations completed in 2018
Tropical Cyclone Pam Recovery Program Evaluation
The evaluation report synthesises a number of complex program and policy matters in an easy to read, well-structured and coherent way. The report uses a range of methods to inform the reader of findings and conclusions, e.g. graphs, text boxes, quotes, photos. There is a clear line of sight between lines of enquiry, findings, evidence, conclusions and recommendations. Although some data to measure the program's impact is not available due to weaknesses in the monitoring system, findings and recommendations are well supported by a good balance of quantitative and qualitative evidence. Another significant strength of this evaluation is its use in informing Australia's subsequent recovery efforts to the Ambae volcano crisis, Vanuatu.
PNG Election Support Program Evaluation
A key strength of this evaluation is the comprehensive and sensitive analysis of the impact of a challenging operating environment on program performance. The methodology is appropriate for the purpose and scope of the evaluation and includes field work in two provinces to assess program performance in different contexts. The report is well-written and clearly structured, and presents credible evidence and findings against each of the evaluation questions. Recommendations logically flow from the findings and identify clear and realistic actions and strategies for responding to the challenging environment through future program designs.
Indonesia Water and Sanitation Hibah Evaluation
This evaluation effectively implements a complex terms of reference, combining a completion review of water activities with a progress review of the sanitation component. The evaluation was guided by an appropriate methodology and an evaluation team that drew on relevant expertise, including specialist advice from within DFAT and the partner government. Findings are supported by a wide range of qualitative and quantitative data in an accessible and engaging format through the use of summary boxes, tables, graphs and photos. There is good integration of qualitative data from key informant interviews and focus group discussions to support findings and conclusions. The evaluation draws on the broader international body of evidence to contextualise the comparative performance of the Hibah. The evaluation also includes a good analysis of program performance in relation to addressing gender issues and provides recommendations for strengthening the approach.
Humanitarian assistance to Myanmar
The Terms of Reference provide a clear description of the purpose, background, timeframes and deliverables. Key evaluation questions and methods are outlined, stating that these will be refined in consultation with the evaluation team. The scope is suitable for the time and resources available for the evaluation. The TORs could be strengthened by outlining the role and responsibilities of the two DFAT officers who are part of the team and by stating that the final evaluation report will be published on the DFAT website.
A clearly presented and structured evaluation plan which covers areas of evaluation use, enquiry, methods, limitations and evaluation team roles and responsibilities well. Annexes provide a matrix of data sources, collection methods and relevant tools against each evaluation question and include interview and focus group discussion guides.
This is a good quality report. The report is clear, well written and structured. Evaluation questions are addressed well. There is a clear upfront statement of findings followed by supporting evidence from multiple sources and/or methods. Data sources are clearly identified including numbered interviews. Recommendations are clearly linked to key findings in table form.
The management response is concrete and provides intended actions and timeframes. It specifies who (DFAT) will take things forward, where appropriate by work area and posts.
Australia's response to El Nino Drought and Frosts in PNG 2015-17 (2017)
This is a high-quality evaluation report. The evaluation provides a solid base of evidence and builds a clear link between questions, evidence, analysis, findings and recommendations. The report draws on a wide range of data, including from M&E reporting from implementing partners and acknowledges where data limitations may affect the findings. The analysis was detailed and used to identify a range of useful lessons that DFAT can apply to better prepare for future drought events.
Pacific Women Shaping Pacific Development: Fiji Country Plan Review (2017)
The evaluation plan presents key elements of an evaluation in a succinct and accessible manner. The evaluation plan demonstrates triangulation well by presenting a matrix clearly showing multiple sources of evidence/methods for each evaluation question. The evaluation plan could have been improved by outlining a sampling strategy and addressing any ethical issues.
This is a well-researched, well-presented report. Some of the strengths include: evidence of change is presented at program, partner and beneficiary levels; findings relevant to specific sub-groups e.g. most vulnerable are included; and there is good analysis of contributing, enabling, constraining factors and lessons learned. The report provides a good analysis of the program's M&E system and recommendations for improving it. Questions about the degree of independence (evaluation was led by staff from the program's regional Support Unit) were counterbalanced by balanced evidence, verified through a collaborative workshop with a range of stakeholders.
Fiji Community Development Program (2017)
This Terms of Reference clearly sets out the background, purpose and scope including a detailed timeline, explanation of expected outputs and team make-up. It attached key documents and refers to the DFAT M&E standards. The evaluation questions are clear, and balanced. The number and type of evaluation questions matches the time available and the team capacity.
A comprehensive evaluation plan which provides sound justification for the methodology proposed and clearly outlines the methods and data sources to be used for each evaluation question in a table format. The evaluation plan includes a clear sampling strategy and a framework to guide analysis of data. There is a comprehensive discussion of limitations, including identification of mitigation strategies. Interview guides are provided in the Annex.
This is a high-quality report, presented in a clear and methodical manner. The use of evidence is rich. Each finding is presented in a balanced way and supported with evidence from a variety of sources. The position of the author is unambiguous. There is good analysis of the risks that the context presents and how these impact on the program's performance. The recommendations are clear, actionable and fit for purpose.
Pakistan Trade and Investment Policy Program (2017)
This is a good example of Terms of Reference for a jointly conducted evaluation. The roles and responsibilities of DFAT and the World Bank staff in the evaluation are clearly outlined. The Terms of Reference clearly state DFAT's evaluation requirements, including the need to publish the report and comply with the DFAT M&E Standards.
A clear, concise report, which is supported by a good stand-alone executive summary and Annexes which provide further details to substantiate findings. Findings are clearly supported by evidence from a range of sources, including data from the M&E system where available. The evaluation addresses an appropriate balance of operational and strategic performance issues. The evaluators clearly state their position about program progress. Conclusions and recommendations flow logically from the findings.
A good example of a management response for a jointly conducted evaluation. The actions that are being taken by both DFAT and the World Bank to address the recommendations are clear, and include additional actions by DFAT to improve program monitoring and performance. Most actions are ongoing, but other actions include specific timeframes for implementation. Clear explanations are provided for why two recommendations were only partially accepted.
Timor-Leste National Program for Village Development (2017)
The evaluation plan's discussion of approach, users, methods for data collection, sampling and analysis are of good quality. The methods/tools to be used for each evaluation question are clear and the Annexes include detailed interview and focus group guides tailored for different stakeholder groups. The evaluation plan could be improved by further discussion of limitations and ethical issues and an allocation of tasks to team members.
Timor-Leste Nutrition Strategic Review (2017)
The evaluation plan has been developed in close collaboration with the primary audience. A comprehensive document which covers all the key elements of an evaluation plan. Limitations, ethical issues and risks are well addressed.
Indonesia Infrastructure Initiative Phase 2 Impact Assessment (2017)
This management response demonstrates how recommendations for an evaluation of an investment which is near completion can be used to inform the design of a new related investment. The management response provides very clear explanations of how the recommendation of the evaluation are being used to inform the new design. Where recommendations are only agreed in part, good justification is provided for adopting a revised approach.
Eastern Indonesia National Roads Improvement Program Review (2017)
The terms of reference clearly and succinctly outline the purpose, background, users, scope and deliverables. The evaluation questions are focused and relevant for a completion evaluation. The roles and responsibilities of the evaluation team and management arrangements with DFAT are well described. The evaluation states that the evaluation will be published on the DFAT website.