ANNEXES

List of Annexes
Annex 1: Consolidated Recommendations
Annex 2: Terms of Reference
Annex 3: List of documents
Annex 4: Itinerary and persons consulted
Annex 5: Compliance with the Paris Declaration on Aid Effectiveness and Accra Agenda for Action
Annex 6: The effectiveness of LJSP in supporting the five goals of the Sector Strategic Framework
Annex 7:  Risks to achievement of program objectives
Annex 8: Impacts from external factors
Annex 9: Chronological Summary of Monitoring and Evaluation in LJSP

Annex 1: Consolidated Recommendations
Recommendation 1: 
(a)	That the PALJP ISP engages with the sector and agencies with a view to undertaking solid needs assessments and in refocusing work plans on increasing service delivery capacities at both the national and sub-national levels;
(b)	In the design of future programs of support, whether for the law and justice sector or for other sectors, AusAID and partner stakeholders take care to ensure that support for strengthening service delivery capacities is given equal attention to support for corporate capacity development.
Recommendation 2: That the NCM, with support from the PALJP, consider
(a)	Undertaking periodic stocktakes to determine whether agencies have the capacities to participate in sector wide activities, to take the lead in formulating policies and implementation plans and in managing change processes; and 
(b)	Using these stocktakes to guide agencies in developing specific initiatives for addressing capacity deficits and identify roles and responsibilities, for funding through the DB.    
Recommendation 3: That the PALJP ISP and AusAID engage in sufficient analytical work to predict change patterns and adjust to changing needs with flexibility.
Recommendation 4:  That the NCM and AusAID consider how the PALJP can support an approach to sectoral planning and budgeting that better integrates the development, recurrent and supplementary budgets.
Recommendation 5: That PALJP ISP supports the sector in developing its capacity to collect meaningful information on its needs and its progress, and to develop systematic and ongoing communication mechanisms to ensure government support.
Recommendation 6: That 
(a)	PALJP ISP engages with sector agencies with a view to supporting them to engage in stakeholder mapping and outreach, and at the same time providing support for partnership development and community engagement capacities;
(b)	PALJP management and advisers be sensitised to look for and recognise emerging issues which may provide entry points for increased civil society engagement, and act promptly to take advantage of the opportunities that such emerging issues may present;
(c)	AusAID considers reserving some of the funding under PALJP to support activities that are both consistent with the National Policy and SSF, but also are not being catered for through the DB process. This could include, in particular, community-based approaches to restorative justice;
(d)	In the design of future sectoral programs (not limited to law and justice) AusAID recognises the risk that reliance on the DB process as the program planning and budgeting  system may result in the neglect of civil society and private sector engagement in the sector, and ensure that future designs include strategies and/or options for mitigating this risk. 
Recommendation 7: That
(a)	The NCM seek assistance through PALJP for a significant focus on increasing agencies’ capacities to identify their target beneficiaries, distinguish these by gender and age group, evaluate their service delivery and develop and implement systems for monitoring and reporting meaningfully on their service delivery to women and men, girls and boys in the community; 
(b)	The key performance indicators of the Sector PMF included in the M&E Framework of January 2008 should be amended as follows: KPI 1: “The police service meets the expectations of women and men in the community” and KPI 4 “All women and men have greater access to justice services.”  
(c)	The NCM  and AusAID should consider how the PALJP can support refinement of the PMF to ensure that its indicators are gender disaggregated wherever possible, and that indicators are designed to measure the service delivery targets as defined by the agencies;
(d)	The indicators to be used by AusAID for assessing the ISP’s performance include the effectiveness of PALJP development practitioners in increasing agencies’ capacities to identify their target beneficiaries, to distinguish these by gender and age group, to evaluate their service delivery and improve existing systems for monitoring and reporting meaningfully on their service delivery to women, men, girls and boys.
Recommendation 8: That
(a)	The NCM and AusAID ensure that PALJP maintains or increases the level of support allocated for gender equality and HIV/AIDS mainstreaming, and continues to support and facilitate the work of the CCI Team;
(b)	The NCM and AusAID use PALJP to support the implementation of the sector’s gender strategy by reviewing and updating the original gender analysis undertaken for the LJSP and using this to drive new and existing initiatives; and 
(c)	As part of the design for any future sectoral program, AusAID requires a thorough gender analysis to be undertaken and documented, linked to baseline data on agency service delivery level, risks, stakeholder analysis, and lessons learned from any relevant previous program.
Recommendation 9: That 
(a)	The PALJP ISP consider applying an action-learning approach based on adult learning principles to its own operations, as well as modelling them for the benefit of agencies, the sector and the M&E system as a whole;
(b)	Future designs for innovations such as sector programs be based on action-learning principles; and
(c)	In Request for Tender specifications AusAID should require tenderers to show evidence of their understanding of and commitment to systematic action-learning approaches in their proposals.
Recommendation 10: That 
(a)	The PALJP ISP provides advisers with ongoing assistance to ensure they are provided with best practice information and alternatives from different jurisdictions; and 
(b)	The sector considers seeking support through PALJP for the creation of a pool of local capacity development trainers that could assist across all agencies, and the development of appropriate training manuals.   
Recommendation 11: That designs and tender specifications for future AusAID programs draw on the strengths of the Assessment Centre approach of LJSP where appropriate.
Recommendation 12: That the NCM, the DNPM and the Department of Justice and Attorney General consider and resolve the future institutional home of the LJSS, and how it can be budgeted. 

Annex 2: Terms of Reference
INDEPENDENT COMPLETION REPORT
LAW AND JUSTICE SECTOR PROGRAM (LJSP) July 2009

AusAID seeks to commission an Independent Completion Report (ICR) of its Law and Justice Sector Program (LJSP) in Papua New Guinea (PNG).  The LJSP is a comprehensive sector program that supports the implementation by the Government of PNG (GoPNG) of its National Law and Justice Policy, sector priorities and sector plan.  AusAID has contracted Cardno ACIL as the Managing Contractor to manage the delivery of support under the LJSP.  The LJSP was completed in April 2009. 
The aim of this ICR is to:
(a)	assess the LJSP sectoral approach and evaluate the performance of the LJSP, with reference to the five DAC criteria for evaluating development assistance: relevance, efficiency, effectiveness, impact and sustainability, and the three additional AusAID criteria of: monitoring and evaluation, gender equality, and analysis and learning;
(b)	enable AusAID and GoPNG to reflect and act on the lessons from the LJSP;
(c)	inform the design and implementation of future assistance to improve AusAID’s ability to meet GoPNG development challenges; and
(d)	build evidence and learning to support AusAID’s Annual Review of Development Effectiveness report, Annual Thematic Performance Reports, Annual Program Performance Reports and Country/regional strategy reviews.
The ICR should be conducted in accordance with AusAID’s guidelines for managing an independent evaluation of an aid activity, which are provided in Attachment C.
1.	Background:
Law and justice is one of seven development priorities identified in the PNG Government’s Medium Term Development Strategy. The sector includes police, courts, justice department, prisons and Ombudsman Commission, as well as a range of other government, non-government and community partners. To support the law and justice sector the Governments of Papua New Guinea and Australia agreed on a sectoral approach to the provision of development assistance. As part of this approach, both governments agreed to measure the performance of the sector as a whole and to assess the relative impact of government and donor funding.
Consequently over the period 2003 to 2009, Australian assistance to the law and justice sector has focused on providing support through three sector-wide programs: 
Through the Law and Justice Sector Program (LJSP), to support the GoPNG to achieve its own policy objectives in the sector, including through funding support, infrastructure and advisers; 
Through the Justice Advisory Group (JAG), to support the sector to develop a performance monitoring framework and increase its ability to use performance information to inform policy decisions and resource allocation; and
Through the Strongim Gavman Program (SGP), formerly the Enhanced Cooperation Program, to support the sector by providing prosecutors, litigation lawyers, and policy and corrections officers to work in key agencies. 
The LJSP, costing AUD 150 million from 2003 to 2009, was the primary program through which Australian financial support to the law and justice sector in PNG has been channelled during this period.   
The explicit purpose of LJSP throughout its six-year term was to develop the capacity of law and justice agencies and their partners to implement the GoPNG National Law and Justice Policy, sector priorities and plans.  To achieve this, the LJSP design provided a flexible program approach through which assistance would be delivered under PNG leadership and, increasingly, using GoPNG systems and processes. In the context of PNG’s evolving policy framework, the LJSP objectives have been adapted and refined over time in order to align with emerging priorities, particularly those articulated in the 2005 Law & Justice Sector Strategic Framework.
In 2003, Cardno ACIL was engaged by AusAID as the Managing Contractor for the LJSP.  The activity has been implemented in 2 Phases from 7 April 2003 to 31 December 2003 (Phase I – Design and Implementation); and from 1 January 2004 to 6 April 2008 (Phase II - Implementation), with Phase II being extended from 7 April 2008 to 6 April 2009.  The LJSP was in operation for six years and ended in April 2009.
2	Objectives of the ICR:
The objectives of this evaluation are to:
2.1	assess the LJSP’s sectoral approach to the delivery of development assistance to the law and justice sector in PNG.  In particular, the technical, institutional and country specific factors influencing the performance of the LJSP should be examined together with the extent to which technical advisers have been used within the LJSP.  Broad comparisons with other law and justice approaches implemented by multilateral and Governmental development agencies should be undertaken to assess the relative effectiveness of the LJSP and to obtain a sharper profile of the strengths and weaknesses of the LJSP’s approach. 
2.2	evaluate the performance of the activity against the criteria of relevance, effectiveness, efficiency, impact, sustainability, monitoring and evaluation, gender equality, HIV/AIDS, and analysis and learning, against which all AusAID activities are required to be measured at completion through the ICR process (evaluation guidance/definitions are provided in Attachment C).  The issues of particular significance in the LJSP activity are:
(a)	the effectiveness of the LJSP in meeting its objectives against the Outcomes and Objectives listed in Attachment D 
(b)	the impact of the LJSP on sector performance and the lessons that can be learned from the LJSP in this regard 
(c)	the efficiency of the implementation of the LJSP, including the performance of the contractor in managing the LJSP activity in meeting contract service delivery objectives as listed in Attachment A
(d)	the sustainability of any improvements in sector performance induced by the LJSP and the factors critical to the continuity of any positive trends in sector performance
(e)	the degree to which gender equality and HIV/AIDS mainstreaming has been integrated into the LJSP 
The ICR evaluation will consider the Activity Completion Report prepared by the LJSP Managing Contractor to establish the ICR parameters, key assumptions and risks in the activity and issues for further investigation during the ICR evaluation.  To assist the evaluation, project documents such as LJSP progress reports (six monthly & annual reports), annual contractor performance assessments, LJSP tasking notes and other relevant LJSP documentation will also be made available to the ICR Team.  These are listed in Attachment B.
3.  ICR Evaluation Team Composition
The ICR will be undertaken by a principal evaluation team of five members consisting of a Monitoring and Evaluation Specialist, a Law and Justice Specialist, an AusAID representative, and a representative of the PNG law and justice sector nominated by the Department of National Planning and Monitoring (DNPM), and a representative of the DNPM.  The team will be led by the M&E Specialist.   
4. Roles and Responsibilities of the ICR Team
(a)	The M&E Specialist will be the team leader and will be responsible for managing, compiling and editing inputs from the other team members to ensure the quality of reporting outputs.
(b)	The Law and Justice Specialist, AusAID representative and the representative of the PNG law and justice sector, and the representative of the DNPM, will work as team members under the overall supervision of the team leader. 
(c)	Subject to (b) above, the role of the Law and Justice Specialist will be primarily focused on addressing objective 2.1 of this TOR and providing overall law and justice related support to the team leader.
(d)	Subject to (b) above, the role of AusAID’s representative will be to provide evaluation support and background information to other team members on the LJSP, other AusAID Law and Justice sector objectives and programs, and contextual information related to AusAID.
(e)	Subject to (b) above, the role of the representative of the PNG law and justice sector will be to provide evaluation support and background information to other team members on the LJSP and contextual information related to the PNG law and justice sector. 
(f)	Subject to (b) above, the role of the representative of the DNPM will be to provide contextual information related to the GoPNG and the PNG law and justice sector.
(g)	The evaluation team will work under the management of the Evaluations Manager, PNG Branch, AusAID Canberra.
5. Process and Approach
The evaluation team will:
(a)	Meet with the AusAID post in Port Moresby before and after the mission for briefing and debriefing.
(b)	Meet the LJSP team leader and staff for an overview of the Activity Completion Report (ACR) with a focus on LJSP’s achievements, lessons learned and broader issues of quality. The team should use this occasion to investigate key performance issues related to LJSP’s functions; management approach; KPIs; HIV/AIDS mainstreaming and gender equality and identify issues for further investigation. 
(c)	Meet with PNG counterparts in the Sector including Department of National Planning and Monitoring (DNPM) and Civil Society Organisation representatives to discuss the purpose of the ICR and gather new and additional data (qualitative and quantitative) to report on the LJSP’s objectives, impact and sustainability.
(d)	Consult AusAID, Sector representatives, National Coordinating Mechanism (NCM) representatives, DNPM representatives, LJSP management personnel and advisers, Community Justice Liaison Unit (CJLU), Law & Justice Sector Secretariat (LJSS) and other stakeholders.   The discussion should focus on the impact of LJSP functions and sector performance development over the life of the LJSP.
(e)	Present an Aide Memoire at the end of the mission to DNPM, PNG Post and key stakeholders.
(f)	Present a seminar/workshop in AusAID, Canberra at a date and venue to be advised by AusAID.
6. Duration
The evaluation is estimated to take four weeks, consisting of three weeks in PNG and one week outside PNG allocated to writing.  The research will require the ICR team members to be in PNG from 20 July to 7 August, 2009.
7. Output
The evaluation team shall submit the following outputs:
(a)	A Draft Methodology for review by AusAID and DNPM prior to commencement (16 July, 2009).
(b)	An Aide Memoire at completion of the mission (7 August, 2009).
(c)	A Draft Report for consideration by AusAID and DNPM within three weeks of the completion of the field mission (4 September, 2009) to the Law and Justice Manager, PNG Branch, AusAID Canberra.  Feedback from AusAID and DNPM will be provided within two weeks of receiving the draft report (18 September, 2009).
(d)	A Final Report for endorsement by AusAID, DNPM and NCM two weeks after feedback (2 October, 2009). 
The evaluation team will prepare a report of 25 pages maximum of text in accordance with AusAID’s Guidelines for ICR reporting.  The structure of reporting should be based on AusAID’s Guidelines for ICR reporting as stipulated in AusAID’s ‘Rules and Tools’ for the ‘Completion and Evaluation of an Aid Activity’. (Guidance / documentation to support the preparation of the ICR will be provided by AusAID).  Lessons and recommendations should be clearly documented in the report. 
Attachment A
Contract Service Obligations - Extract from Contract 11622 Amendment 8 - Schedule 1 Part A1 Scope of Services
4.	CONTRACTOR’S GENERAL OBLIGATIONS
4.1	Design and implement
The Contractor will, in consultation and co-operation with the GoPNG and AusAID, design and implement a program of activities to achieve the Goal, Purpose, Outcomes and Objectives contained in Clause 2 above and in accordance with the requirements of this Part A1 General Requirements and Parts A2 - Design Requirements; Part A3 - Implementation Requirements, and Part B. Implementation of Interim Program Plan Phase I; Part C. Implementation of Annual Program Plan 2004; Part D. Implementation of Annual Program Plan 2005; Part E. Implementation of Annual Program Plan 2006; Part F. Implementation of Annual Program Plan 2007; Part G. Implementation of Annual Program Plan 2008 below.
4.2	Mobilisation
(a)	The Contractor will facilitate a nine-day planning workshop with Design Team members in Australia commencing on the Program Start Date.
(b)	The Contractor must mobilise the Team Leader and administrative support staff in PNG on the Program Mobilisation Date. 
(c)	The Contractor will establish a Program Office in Port Moresby, and communicate its contact details, along with the purpose of the Program, to all relevant Stakeholders.
4.3	Program management
The Contractor will manage the Program, including resources, personnel and costs, to achieve the agreed outcomes and outputs within the time and budget set, in a culturally sensitive and appropriate manner:
(a)	in Phase I, in accordance with the agreed Program Design Framework workplan, Program Design Framework and Interim Program Plan;
(b)	in Phase II, in accordance with the agreed Program Design Document, Annual Program Plans and Agency Plans; and
(c)	at all times, in accordance with all relevant terms and conditions of the Contract.
4.4	Adviser recruitment
(a)	The Contractor must:
(i)	ensure that all Adviser recruitment is conducted in accordance with the Commonwealth Procurement Guidelines (CPGs), including the core principle of “value for money” (incorporating both technical and price assessments) and all supporting principles, and all relevant regulatory requirements and AusAID policies;
(ii)	engage Advisers against the approved Terms of Reference for the position;
(iii)	provide an overall quality assurance role with respect to all Program personnel, including (but not limited to) quality control in relation to: any reports/documentation prepared by Program personnel and performance of individual Advisers in relation to specific tasks;
(b)	In recruiting Advisers for the Program, the Contractor must use a variety of methods, as appropriate.  These may include (but are not limited to) the following options: direct advertising in national daily/weekend newspapers and/or professional journals, use of specialist recruitment agencies, use of the company's website and its database to alert consultants to the opportunity.  
However:
(i)	except where justifiable on value for money grounds taking into account factors including the position and level of inputs, the Contractor must not limit advertising to the use of its own database;
(ii)	except where justifiable on value for money grounds taking into account factors including the position and level of inputs, the Contractor must ensure that all Adviser positions are advertised in private and public sector fora;
(iii)	the Contractor shall use its best endeavours to engage PNG nationals wherever appropriate; and
(iv)	the Contractor will ensure GoPNG participation in interviews for recruitment of LTAs and STAs, as appropriate.  The Contractor must notify AusAID of proposed GoPNG participation in those selection exercises in which it is considered appropriate, prior to the proposed involvement.
(c)	Prior to engagement of any Adviser the Contractor must provide AusAID and GoPNG with a copy of the preferred candidate’s CV.
(d)	Where requested by AusAID, the Contractor shall submit to GoPNG and AusAID a copy of the preferred candidate's application with a brief report (approximately one page) on the recruitment strategy, interview and referee checks, together with any other assessments used (which must include price), and a comment on any alternate short-listed candidates.  AusAID reserves the right to reject the Contractor’s preferred candidate at AusAID’s absolute discretion and to require the Contractor to undertake further recruitment activities at the Contractor’s expense.
4.5	Program monitoring and reporting
The Contractor will monitor the timely achievement of the Program outcomes and outputs against verifiable indicators, in accordance with this Contract and the agreed Program Design Framework and Program Design Document.  This will include submission of reports as per Clause 6 of Schedule 1, Scope of Services, Part A3 (Implementation Requirements).  The Contractor will take appropriate action to address shortcomings or review its approach where necessary.
4.6	Cooperation with JAG personnel
(a)	The Contractor will provide to JAG personnel and the JAG secretariat every assistance in its monitoring role and in relation to the ongoing development of sector performance information (refer, in particular, Clause 11 of the Program Specific Contract Conditions). 
(b)	The Contractor must ensure a productive and professional working relationship with the JAG, which may include, variously, the facilitation of meetings with GoPNG officials and other stakeholders, and the provision of documentation and verbal briefings.
4.7	Security Plan
The Contractor will be responsible for the immediate development and implementation of a Security Plan in accordance with Clause 21.1 of the Program Specific Contract Conditions.  The Contractor must revise and update the Security plan annually.
4.8	Construction Work
(a)	The Contractor must manage the procurement and delivery of Construction Works:
(i)	within the financial limit approved by AusAID for the particular package(s) of Construction Works; and
(ii)	in accordance with this Clause 4 and the terms and conditions of the Contract (refer, in particular, Clauses 12 and 13 of the Program Specific Contract Conditions).
(b)	The Contractor shall submit to AusAID for approval a written proposal in relation to each package of proposed Construction Works, which shall contain:
(i)	detailed costings and technical specifications for the construction output(s) to be delivered;
(ii)	a specific timeframe for delivery; and
(iii)	any other information, as requested by AusAID.
(c)	AusAID will not accept any liability whatsoever (including any loss or costs incurred a third party) for payment in relation to Construction Works commenced without AusAID’s prior, specific written approval.
(d)	The Contractor shall undertake the roles of project manager and FIDIC Engineer (or equivalent) in relation to the Construction Works. Where AusAID has required the Contractor to sub-contract design work in relation to the relevant Construction Work, the Contractor shall undertake the role of project manager and the design sub-contractor shall undertake the role of FIDIC Engineer (or equivalent). In these circumstances, the Contractor’s responsibilities as project manager include managing the relationship between the design and construction sub-contractors. In all cases, the Contractor shall undertake the full range of project management responsibilities.  
(e)	Payment of construction sub-contractors shall occur:
(i)	through the Sub Contractor’s Imprest Account (refer Clause 23 of the Program Specific Contract Conditions), in which case the Contractor must submit an Unconditional Financial Undertaking in accordance with Clause 24 of the Program Specific Contract Conditions; or
(ii)	on a reimbursable basis, at cost, in arrears, in accordance with Clause 10, Part A2 of Schedule 2 (Basis of Payment), in which case Clause 24 of the Program Specific Contract Conditions shall not apply.
(f)	The Contractor shall not be entitled to any separate or additional supervision fee for the supervision and management of any Construction Works under this Contract.  (Construction management costs have been included in the Contractor’s Total Management Fee, payable in accordance with Clause 2 of Schedule 2, Part A2 (Basis of Payment)).
4.9	Quality Assurance
(a)	The Contractor is responsible for ensuring that all Program functions and operations for Phases I and II are achieved efficiently, to the highest professional standards and in a manner consistent with AusGuide.  Within 3 months of the Program Mobilisation Date the Contractor will provide AusAID with the following Quality Assurance documents: 
(i)	Communications Plan; and the
(ii)	Program Administration Manual.
(b)	As proposed by the Contractor, the Contractor will utilise the services of a Technical Advisory Support Panel (TASP) as a Quality Assurance reference group. The TASP shall be utilised by the Contractor in an internal Quality Assurance capacity only. Beyond this the TASP does not have a formal Quality Assurance role in the program and shall not: 
(i)	undertake any form of Program implementation role; 
(ii)	perform an external Program or Sector performance monitoring, evaluation or assessment role; or 
(iii)	interfere in any way with the performance by the JAG of its Sector and Program performance monitoring, evaluation and assessment role.
(c)	Individual TASP members may, subject to the requirements of Adviser Recruitment in Clause 4.4, Part A1 of this Schedule, undertake program implementation work.  In such circumstances the individual shall cease to be a TASP member for the duration of the implementation work, and an appropriate replacement to the TASP shall be made by the Contractor.

Attachment B 
Background Documents for LJSP Review, July 2009
A. Contract/Foundation Documents
1.	Law & Justice Assistance and Poverty Reduction in PNG, Desk Study 2002 
2.	Concept Paper for a Law and Justice Sector Program November 2002
3.	LJSP Program Design Framework
4.	LJSP Program Design Document (Volumes I and II)
5.	LJSP Contract – Scope of Services (as at latest amendment)
6.	LJSP Monitoring & Evaluation Framework
B.  LJSP Annual Program Plans
7.	LJSP Interim Program Plan 2003
8.	LJSP Annual Program Plan 2004
9.	LJSP Annual Program Plan 2005
10.	LJSP Annual Program Plan 2006 
11.	LJSP Annual Program Plan 2007
12.	LJSP Annual Program Plan 2008
C.	LJSP Reports
13.	Six month report (Jan-June 2004)*
14.	Six month report (Jan-June 2005)*
15.	Six month report (Jan-June 2006)*
16.	Six month report (Jan-June 2007)*
17.	Six month report (Jan-June 2008)*

18.	Six month/Annual report (Apr-Dec 2003)
19.	Six month/Annual report (Jul-Dec 2004)
20.	Annual report (Jan-Dec 2005)
21.	Annual report (Jan Dec 2006)
22.	Annual report (Jan-Dec 2007)
23.	Annual report (Jan-June 2008)

24.	LJSP Activity Completion Report
D.  LJSP Contractor Performance Reviews
25.	Annual Contractor Performance Assessment 2003
26.	Annual Contractor Performance Assessment 2004
27.	Annual Contractor Performance Assessment 2005
28.	Annual Contractor Performance Assessment 2006
29.	Annual Contractor Performance Assessment 2007
30.	Annual Contractor Performance Assessment 2008
AusAID Response to LJSP Contractor Performance Reviews
31.	AusAID Response - Annual Contractor Performance Assessment 2003
32.	AusAID Response - Annual Contractor Performance Assessment 2004
33.	AusAID Response - Annual Contractor Performance Assessment 2005
34.	AusAID Response - Annual Contractor Performance Assessment 2006
35.	AusAID Response - Annual Contractor Performance Assessment 2007
36.	AusAID Response - Annual Contractor Performance Assessment 2008
Other AusAID LJSP Contractor Performance Reports
37.	Quality at Entry Report 2004
38.	LJSP Activity Monitoring Brief 2004
39.	LJSP Simplified Monitoring Toolbox 2005 
40.	LJSP Simplified Monitoring Toolbox 2006
41.	Quality at Implementation Report 2006
42.	Quality at Implementation Report 2007
43.	AusAID 1-page Contractor Performance Assessment Report 2003
44.	AusAID 1-page Contractor Performance Assessment Report 2004
45.	AusAID 1-page Contractor Performance Report 2005
46.	AusAID 1-page Contractor Performance Report 2006
47.	AusAID 1-page Contractor Performance Assessment Report 2007
48.	AusAID 1-page Contractor Performance Assessment Report 2008
E.  Justice Advisory Group Reviews and Analysis
Restorative Justice
49.	Application and Implications of GoPNG’s Law and Justice Policy and Plan of Action, as it relates to Restorative Justice - Assessment, April 2004*
50.	Working Paper on Restorative Justice and Community-Oriented Approaches to Crime Prevention and Dispute Resolution, July 2004*
Sector Facilities
51.	Law & Justice Sector Program Facilities Review and Strategy Development Mission – Review Report, January 2006 
52.	Sector Facilities Strategy Recommendations, January 2006 
53.	LJSP Facilities Technical and Financial Audit, November 2006
Community Justice and Crime Prevention
54.	Yumi Lukautim Mosbi Impact Evaluation 2006 Progress Report*
55.	PNG Village Courts Support Review, April 2007*
56.	Community Justice Liaison Unit Review, November 2007* 
Sector Budget Process
57.	Review of the 2008 Law and Justice Sector Development Budget Process, November 2007*
Other Reports
58.	JAG Project Director Mission Reports (various 2003-2008)*
F. AusAID Reports and Commissioned Reviews & Analysis
59.	PNG Law & Justice Sector – Review and Contribution Analysis December 2006 (AusAID)
60.	AusAID Assistance to PNG’s Law and Justice Sector (2003-2007): Lessons Learned – Desk Review 2007 (Armytage)*
61.	PNG Law & Justice Sector – Approaches to Capacity Building 2007 (Lyon)
62.	AusAID Assistance to PNG’s Law and Justice Sector (2003-2007): Monitoring & Evaluation 2007 (Kenway)
63.	AusAID assistance to PNG’s L&J Sector 2003 – 2007
64.	Rapid Review of Ombudsman Twinning Program 2008 (Lyon)*
65.	AusAID PNG Program Gender Stocktake (2008)
66.	Office for Development Effectiveness report Violence Against Women in Melanesia and East Timor: A Review of International Lessons 2008
67.	AusAID Annual Sector Report 2006
68.	AusAID Annual Sector Report 2007
G.  GoPNG Law & Justice Sector Policies, Strategies, Reports and Studies
69.	White Paper on Law and Justice, 2007
70.	National Law & Justice Policy & Plan of Action 2000 and Sector Strategic Framework 2005

71.	Law & Justice Sector Working Group, Law & Justice Sector Gender Strategy, 2005*
72.	Law & Justice Sector Working Group, Provincial Engagement Framework, January 2006*
73.	Law & Justice Sector Working Group, Infrastructure Strategy Study Report, March 2002*

74.	Law & Justice Sector Secretariat, Law & Justice Sector Imprest Account Procedures Manual, 2004 (as updated)*

75.	PNG Law & Justice Sector Annual Performance Report 2004 
76.	PNG Law & Justice Sector Annual Performance Report 2005
77.	PNG Law & Justice Sector Annual Performance Report 2006
78.	PNG Law & Justice Sector Annual Performance Report 2007

H.  Additional Documents

79. Draft Baseline Collation Report
80. PNG JAG ICR Final Draft Report
81. JAG Final Activity Completion Report
82. Final LJSP CPA Report

Attachment C
                                         This Independent Completion Report template  (#155) 
                                      is current to 30 November 2009
delete this, and all explanatory text (blue-gray) before the report is finalised
< Title Page >

Aid Activity Name

AidWorks Initiative Number

INDEPENDENT COMPLETION REPORT

Author’s Name and Organisation

Date (month year)

< NOTE: The report should be no more than 25 pages (excluding the annexes) >

Aid Activity Summary
< To be completed by the AusAID evaluation manager before template is provided to evaluation team. >
Aid Activity Name		
AidWorks initiative number		
Commencement date		Completion date		
Total Australian $	< AusAID and other Australian government contribution >	
Total other $	< eg, including amount contributed by other partner donors, partner governments, etc >	
Delivery organisation(s)		
Implementing Partner(s)		
Country/Region		
Primary Sector		

Acknowledgments

Author’s Details

Executive Summary
< This should be a maximum of 2 pages, and be comprehensible as a stand-alone document. The main audience for the executive summary is senior managers and implementing partners.
The executive summary should provide the following information:
>	Background and context (where directly relevant to the findings).
>	A summary of the activity objectives, components and key results.
>	A brief outline of the evaluation findings.
>	A brief outline of the lessons and recommendations.
>	Evaluation Criteria ratings (as below). >

Evaluation Criteria Ratings
< Copy from the rating summary in the main body of the document. >
Evaluation Criteria	Rating (1-6)	
Relevance		
Effectiveness		
Efficiency		
Sustainability		
Gender Equality		
Monitoring & Evaluation		
Analysis & Learning		
Rating scale: 6 = very high quality; 1 = very low quality. Below 4 is less than satisfactory.
Introduction
Activity Background
< Provide information about the objectives, design and implementation history of the activity. Include relevant information on the country context of the activity, and how the activity fits into the country and/or sector strategy. >

Evaluation Objectives and Questions
< Describe the evaluation objective(s) and questions, as defined in the Terms of Reference. >

Evaluation Scope and Methods
< Outline the methods of the evaluation, including sources of evidence and types of analysis used to answer the evaluation questions, duration of evaluation, etc
Outline any assumptions made by the evaluation team and limitations of the methods. >

Evaluation Team
< Brief description of the composition of the evaluation team. Outline any team member’s conflict of interest (such as previous involvement in the activity) and strengths and weaknesses of the composition of the team (such as skills mix, size of the team, etc.).  >

Evaluation Findings
< The main body of the report should directly answer the evaluation questions, as defined in the Terms of Reference. Quantitative and qualitative evidence to support findings and recommendations needs to be presented as part of the report; referring to annexes or other documents is not sufficient. Where possible, data should be disaggregated by sex.
The report structure will be determined by the evaluation questions, and can be adjusted accordingly. Regardless of the structure, findings must specifically address AusAID’s evaluation criteria: relevance, effectiveness, efficiency, impact (if feasible), sustainability, gender equality, monitoring & evaluation and analysis & learning. Assessment of cross-cutting issues and compliance with the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action should be integrated into the evaluation criteria.
Note: further information can be provided in annexes to the main report. At a minimum the Terms of Reference should be provided as an annex. The evaluation plan could also be provided. >

Relevance
< To determine whether the activity contributed to higher level objectives of the aid program (outlined in country and thematic strategies). >

Effectiveness
< To determine whether the activity has achieved its objectives. >

Efficiency
< To determine whether the activity was managed to get the most out of the inputs of funds, staff and other resources, including continual management of risks. >

Impact
< To determine whether the activity has produced positive or negative changes (directly or indirectly, intended or unintended). The degree to which the various aspects of impact can be assessed will vary according to the nature and duration of the activity. Whether impact can be assessed, or the way impact can be assessed will need to be determined by the Independent Evaluation Team. Impact will not be rated. >

Sustainability
< To determine whether the activity has appropriately addressed sustainability so that the benefits of the activity will continue after funding has ceased, with due account of partner government systems, stakeholder ownership and the phase-out strategy. >

Gender Equality
< To determine whether the activity advanced gender equality and promoted women (considering the four dimensions of gender equality: access, decision-making, women’s rights, capacity-building). >

Monitoring and Evaluation
< To determine whether the activity's monitoring and evaluation system effectively measured progress towards meeting objectives. >

Analysis and Learning
< To determine whether the activity was based on sound technical analysis and continuous learning. >

Evaluation Criteria Ratings
< AusAID requires that the author(s) rate the quality of the aid activity based on the evaluation criteria (excluding impact).  This section should be no more than one page. >

Evaluation Criteria	Rating (1-6)	
Relevance		
Effectiveness		
Efficiency		
Sustainability		
Gender Equality		
Monitoring & Evaluation		
Analysis & Learning		
Rating scale:
Satisfactory	Less that satisfactory	
6	Very high quality	3	Less than adequate quality	
5	Good quality	2	Poor quality	
4	Adequate quality	1	Very poor quality	

Conclusion and Recommendations
< The conclusion should draw together implications of the findings and provide an overall assessment of the quality and success of the aid activity.
Specific lessons for the program and the design of new activities should be identified. Lessons can either have broad value across a range of sectors, or be specific to the particular sector, theme or country.  They should avoid generic statements, and where possible, provide new insights into how AusAID can do things better in future.  Lessons need to be clear, specific, actionable and supported by the analysis in the report.
Where recommendations are made, these should be directly discussed with AusAID program staff so they are appropriately informed by program priorities and constraints.>

Attachment D
Attachment D – LJSP Management and Outcome Indicators

TABLE 1: MANAGEMENT INDICATORS

Program Governance Indicators

Output	Key Indicators	Information Sources	
Financial Management 
Financial management systems in place and able to provide AusAID with routine and ad hoc requests for financial information	
Accurate financial information and invoices provided to AusAID in a timely manner

AusAID satisfied with financial information received

LJSP proactive in identifying significant variations on expenditure and advising AusAID with a proposed response

Finances managed in accordance with contractual and statutory requirements	
AusAID feedback – contractor performance reports

Audits of LJSP	
Personnel 
Effective personnel recruitment, mobilisation and management systems in place	
Appropriate personnel recruited, briefed and mobilised

AusAID and GoPNG satisfied that personnel performance is being managed effectively
	
Adviser performance appraisals
Adviser recruitment schedules and spreadsheet
Adviser feedback
AusAID feedback
GoPNG feedback	
Communications
Open and effective communications maintained with key PNG stakeholders, AusAID and the JAG	
AusAID, JAG and key PNG stakeholders express satisfaction with communication processes

Significant issues likely to impact on achievement of program goal and objectives and proposed responses identified and communicated to AusAID in proactive manner

Timely and accurate proposals for contract variations or amendments provided to AusAID	
PNG stakeholder feedback
AusAID/JAG feedback
Cardno ACIL/LJSP feedback	
Reporting
Quality LJSP reports produced in a timely manner	
Reports submitted on time

Reports judged to be appropriate standard and quality	
AusAID/JAG/GoPNG feedback	
Procurement and Subcontracting	LJSP undertakes procurement and subcontracting in accordance with GoPNG requirements and contractual requirements	Procurement documentation	

Strategic Management Indicators

Output	Key Indicators	Information Sources	
LJSP provides AusAID with timely and accurate contextual analysis and policy advice	AusAID confidence in advice from LJSP remains positive	AusAID feedback	
LJSP provides AusAID with rationale and justification for Program position and mix of inputs	Justification and rationale documented to the satisfaction of AusAID

Evidence of M&E implementation shaping mix and focus of inputs	Reviews, audits etc

APP analysis

M&E Framework

AusAID feedback	
LJSP develops and maintains effective relationships with AusAID, JAG and key PNG stakeholders	Satisfaction of key stakeholders with LJSP	Adviser performance appraisals

Stakeholder surveys

Informal feedback

PCG feedback

Reviews	
Clearly defined accountabilities and responsibilities to support LJSP processes
	LJSP team clear about their roles and responsibilities and acting accordingly	PCG/AusAID/Agency/JAG feedback

Stakeholder surveys

Documentation on LJSP roles/responsibilities/processes	
PDD principles are reflected in LJSP actions	Satisfaction of key stakeholders with LJSP way of working	Adviser performance appraisals

Stakeholder surveys

Informal feedback

PCG feedback	
LJSP Strategies for cross cutting issues of gender, HIV/AIDS and capacity building are developed and implemented 	Evidence that strategies are  implemented, monitored and refined based on implementation experience	Adviser work plans

LJSP reports

M&E Framework

Learning and Innovation Indicators

Output	Key Indicators	Information Sources	
LJSP has in place processes promoting learning and for the dissemination of innovation	Adviser work plans are regularly reviewed and revised

Lessons learned are captured in SMR and APP

Evidence of sharing of experience through contacts

Evidence of innovative practices that are shaped by lessons learned	Adviser work plans 

Activity completion reports

SMR analysis

Review of published material

M&E Framework	

TABLE 2: Outcome Indicators
Assessment of outcomes will involve assessment against the LJSP Program Logframe, against 10 of the 11 purpose level indicators as outlined in the JAG’s methodology (Attachment B) for outcomes assessment and agreed to by AusAID and LJSP.

PNG-AUSTRALIA LAW AND JUSTICE SECTOR PROGRAM: LOGICAL FRAMEWORK

Narrative Summary	Performance Indicators	Verification Tools	Assumptions	
Goal				
To move towards a more just, safe and secure society for all people in PNG.				
Purpose				
To support the implementation by GoPNG of its National Law and Justice Policy, through the Sector Strategic Framework. 
	Improvements against SSF objectives, sector and agency indicators as recorded in the Program MEF.

Public and key stakeholders express increasing satisfaction with quality of law and justice services and agencies.

Improved reflection of SSF in all sector and agency planning documents and increasing prioritisation of resource allocations.

Coordination, research and process improvement contributing to increasing system throughput for serious crime.

Increased sector capacity to detain, integrate and reintegrate offenders

Provincial engagement processes working and ‘pilot’ activities successfully implemented.

Community based activities replicated and reports indicating increased satisfaction with community based crime prevention and restorative justice.

Enhanced institutional capacity to investigate and prosecute fraud and corruption.

Sector agencies and civil society designing, implementing and monitoring gender equality, family and sexual violence HIV and AIDS responses.

Enhanced public administration and compliance with government systems and processes in planning, human resource management, finance and reporting.

Strengthened sector coordination.	MEF verification tools as specified.

Sector/JAG surveys.

Sector annual and quarterly reports.

Agency plans.

Agency annual reports.

Activity reports.

Program and adviser reports.

Stakeholder workshop reports.

	Leadership, policy and strategic direction from NCM and government, remains stable and consistent.

Continued support for program approach and increasing sector coordination.

Sector leaders comfortable about cooperation and their ‘independence’ is not threatened.

Adequate funding for sector agencies.

Stability in agency senior leadership and no political interference.

Support from central agencies for sector approach, reforms and activities.

Civil society engages with the sector at all levels.

	
The Result areas below are intended as a convenient higher level summary of the relevant activities as contained in the 2007 Annual Program Plan and LJSP MEF	The performance indictors are intended to be the same as those shown in the individual activity sheets contained within the 2006 LJSP MEF, however, not all such indicators are listed.  In some cases a general indicator is described which is made up of the individual elements listed in the MEF.	Specific verification tools for each LJSP activity are contained in the MEF. Applicable as appropriate in most cases:
o	Agency annual plans & reports
o	Sector performance reports
o	Community Crime surveys Media stories		

SECTOR GOAL 1: IMPROVED POLICING, SAFETY AND CRIME PREVENTION	
Narrative Summary	Performance Indicators	Verification Tools	Assumptions	
Result 1.1: 
Provincial and local mechanisms to improve safety and prevent crime strengthened and developed.

Activity 1.2.1 (EHP)
Activity 1.2.2 (Bougainville L&J)
Activity 1.3.2 (YLM)
Activity 5.3.2 (Provincial Engagement)	Administrations (ABG, EHP, NCDC and select others) planning, coordinating (with law and justice agencies and civil society) and implementing agreed priority activities using participatory processes.

Packages of prioritised support for crime prevention and restorative justice implemented, including increasing provincial support for L&J activities.	Administration and activity plans, reports, including law and justice committee minutes.

Provincial L&J plans and budgets
Activity progress reports.	Administration committed to staffing and resourcing law and justice positions and activities.

Functioning local law and justice coordination mechanisms involving formal and informal sectors.	
Result 1.2:  
Policies, practices and programs to protect children and women’s rights functioning effectively.

Activity 1.3.1 (Reduce family violence)	
Family and sexual violence programs established and supported by agencies and civil society to implement the Family and Sexual Violence Strategy.

Policies, practices and processes trailed and extended to support victims and witnesses.

Research completed and disseminated.	
Activity progress reports.

Documented processes

Research reports.	Support from CIMC, FSVAC CJLU, Department of Justice and Attorney General (DJAG), PP, PS, Police, UNICEF and others to adopt a cross-sector cooperative approach. 

Agencies resourced to support the protocols in daily operations.	

SECTOR GOAL 2: INCREASED ACCESS TO JUSTICE AND JUST RESULTS	
Narrative Summary	Performance Indicators	Verification Tools	Assumptions	
Result 2.1: 
Sustainable processes for efficient handling of cases by each agency developed and maintained.

Activity 2.1.1 (Processing serious crime)
Activity 2.1.4 (District Court processes)
Activity 2.1.5 (National Court)
(nb: these two activities replace 2006 APP Activity 2.1.2)

	Formal agencies and civil society working together to improve court processes.

Judiciary leading the review, design and implementation of the reengineered processes.

Completion and implementation of new indictable case stream: clearance of backlog; reduction in time taken to process cases; reduction in multiple adjournment; improved case flow; improved user perception of serious crime processing

Agency case management systems contributing to improved work flow in each agency.

Improved case flow and improved user perception of access, fairness and equality in National and District Court civil processes, especially registry practice.

	Court User Forum minutes

Court User Forum Minutes. Correspondence with Chief Justice.

Task Force Report.
Research data base outputs.
Court User Forum Reports.

Case management system policies and manuals/handbooks
Agency case flow reports

Case management systems working

	Current levels of enthusiasm and participation continue

Chief Justice continues to support Court User Forum approach: justice select ‘drives’ the process

Judges, magistrates and senior lawyers committed to reforms.

All agencies committed and resourced to implement their component of the reform

Senior managers and agencies committed to ensuring systems are used

Result 2.2:  
Systems, practices and infrastructure for improved access to justice and fair outcomes strengthened.
Activity 2.2.2 (Village Courts) including  2006 APP  Activity 2.2.1
Activity 2.1.6 (Facilities & Infrastructure) including 2006 APP Activity 2.1.3 (Bougainville court infrastructure)t
Activity 2.1.8 (Public Solicitor)
Activity 2.1.9 (DJAG)
Activity 2.3.1 (Publishing laws)	
Improved capacity of provincial administrations and the DJAG secretariat to manage & support village courts  

Assets maintenance and procurements contributing to improved court efficiency.

Improved capacity of Public Solicitor, Legal Training Institute and DJAG to provided core functions and services

Sector libraries, law reports, judicial decisions and legislative databases sustained and providing greater access to laws	
DJAG and JAG surveys.

Facilities maintained and systems procured and installed.

Management reports, CMS data, stakeholder surveys

User (judges, lawyers, NGOs) surveys.	
Provincial administrations recognise village courts as a priority and pay allowances.
Agency priority plans actually used and ad hoc spending decisions eliminated.

Judges make decisions available for publication.	
Result 2.3:  
Restorative justice engagement models piloted and promising approaches replicated to strengthen communities to maintain peace and good order.

Activity 2.2.1 (Restorative Justice) – village courts now included in Activity 2.2.2
Activity 2.1.9 (DJAG) – part for CJC	
New restorative justice activities are happening pilots established, scaled up and replicated; community justice centres working, use of mediation in village courts  increasing	
Agency and CJLU reports.	
DJAG and sector AMT continues to take the lead in advancing restorative justice approaches.	

SECTOR GOAL 3: IMPROVED RECONCILIATION, REINTEGRATION AND DETERRENCE	
Narrative Summary	Performance Indicators	Verification Tools	Assumptions	
Result 3.1: 
Policies and practices to promote fair and equitable justice for young people; rehabilitation for detainees; and alternatives to prison strengthened and developed.

Activity 3.1.1 (Juvenile Justice)
Activity 3.2.1 (Community Based Corrections)
	National Juvenile Justice Policy implemented with packages of support in selected centres; diversion programs established in selected centres; community based alternatives to detention established in selected centres.

CBC regional and provincial offices working with NGOs and CBOs to increase support to Community Work Programs; detainee release programs improved in select areas.
	DJAG CBC Director’s Reports.

Provincial Juvenile Justice Working Group reports. 

CBC Action Plan.

LJSP reports.
		
Result 3.2:
Management and operation of correctional institutions strengthened.

Activity 3.3.1 (Prison Operations)
Activity 3.3.2 (Welfare &Rehab)
Activity (3.3.3 (Bougainville) from 2006 now absorbed in Activity 3.3.4 (CS Facilities & infrastructure)	Improved prison operations resulting from refresher training and responses to operational audits

Industry work programs and rehabilitation activities being developed and implemented.

Assets maintenance and new infrastructure contributing to improved security and humane containment of detainees.	Training reports, Operational audit reports

Prison industry data.

Commander reports.

Facilities built and maintained, and systems procured and installed.	CS makes staff and trainers available.

CS managers initiate and manage prison industries and welfare activities.

CS facility plans used and ad hoc spending decisions eliminated.	

SECTOR GOAL 4: IMPROVED ACCOUNTABILITY AND REDUCED CORRUPTION	
Narrative Summary	Performance Indicators	Verification Tools	Assumptions	
Result 4.1: 
Capacity and processes for agencies and the sector to address reduce fraud and corruption strengthened and developed.

Activity 4.1.1 (Prosecution of Fraud & Corruption)
Activity 4.1.2  (F&C Initiatives)
Activity 4.1.3 (Public Prosecutor)	Ombudsman Commission processes for complaint and leadership investigations are strengthened resulting in throughput targets achieved; success rate in referrals and prosecutions increasing.

Leadership tribunals and prosecutions for serious crime taking place as scheduled

NACA established and functioning.

Integrity review recommendations resulting in improved agency systems and processes	Ombudsman plan.

Ombudsman Commission Annual Report.

Twinning partner’s reports.

Training records.

Reports of NACA Chair.	Ombudsman and other constitutional officers cooperate.	

SECTOR GOAL 5: IMPROVED ABILITY TO PROVIDE LAW AND JUSTICE SERVICES	
Narrative Summary	Performance Indicators	Verification Tools	Assumptions	
Result 5.1: 
Ability of key government justice sector institutions to adopt good public administration practices and to use resources properly strengthened and developed. 

Activity 1.1.1 (Police Administration)
Activity 5.1.1 (Personnel Management)
Activity 5.1.2 (Finance & Budget)
Activity 5.1.3 (Corporate Planning & M&E))
Activity 5.1.4 (Facilities & Assets)
Activity 5.1.5 (Information & IT)
Activity 5.1.6 (Reporting & Monitoring)
Activity 5.1.7 (Gender)	Skills development packages delivered in core areas of public administration (especially finance and budgeting).

Capacity for planning enhanced in all agencies with costed annual and corporate plans completed showing linkages to government and sector policies and priorities.

Accountability of agencies increased (e.g. annual reports published; activity completion reports; integrity reviews completed; fraud control plans in place).

Agency facilities and assets management and facilities maintenance strategies being used to determine priorities and guide spending decisions.

Sector Gender Strategy implemented: gender mainstreamed into agency planning and operational practice; gender work place plans in each agency.

	Training packages and delivery records.

Annual and corporate plans, agency planning procedure documents, budget proposals

Annual & quarterly reports. Activity completion reports.  Evaluation reports.

Integrity reviews. Fraud control plans.

Asset & facilities plans & budgets for agencies

IT and IM policies, plans and budgets. Internal user surveys.

EEO and HR policies. Gender data in planning and reporting documentation		
Result 5.2: 
Engagement models piloted and promising approaches replicated to strengthen civil society engagement in the sector.

Activity 5.2.1 (Civil society engagement)	Strengthened linkages between sector agencies and civil society support policy dialogue and service delivery.

Sustainable engagement models and pilot activities are designed, delivered (and existing ones supported) and replicated.

CJLU operating in accordance with agreed strategy and annual plan.	CJLU Operational Strategy.

CJLU Annual and other reports.

Activity reports.

Audit reports.	Civil society willing to participate and has access to adequate resource and leadership.	
Result 5.3: 
Strengthened capacity and processes for sector-wide coordination and implementation.

Activity 5.3.1 (Sector coordination)	Enhanced NCM, LJSWG, LJSS and AMT capacity to lead development of sector planning and implementation (e.g. successful planning and delivery of cross sector multi-agency activities; sector presentations and reporting to government; budget allocations reflecting SSF).

Increasing LJSS capacity to support and lead sector coordination and the operation of sector mechanisms including the imprest account arrangements.	Recurrent and Development Budget.

Agency annual reports.
NCM, LJSWG, AMT minutes.

LJSS Reports to LJSWG and NCM.

Sector and central agency reports to CACC and NEC.	NCM members individually remain committed to sector approach.

Agencies provide human resources for LJSWG, AMTs and their work.

Government supports sector and programmatic approaches.	

Result 5.4: 
Effective sector and agency strategy and responses to prevent HIV and counteract human resource losses to HIV and AIDS.

Activity 5.4.1 ( HIV & AIDS Response)	Sector and agency HIV/AIDS response evolving within National Strategic Plan framework.

HIV and AIDS mainstreamed into agency planning and operational practice.

HIV and AIDS work place plans operational in high risk areas, especially CS and police

AIDS Work taking place in select key high risk operational areas.

	HIV/AIDS plans and activities.

Agency budgets and annual plans.

Agency annual reports
National AIDS Council Annual Report.	Sector and agency leadership for significant HIV/AIDS response.

Agencies have capacity to address the impact of HIV/AIDS.	

Annex 3: List of documents
A. Contract/Foundation Documents
1.	Law & Justice Assistance and Poverty Reduction in PNG, Desk Study 2002 
2.	Concept Paper for a Law and Justice Sector Program November 2002
3.	LJSP Program Design Framework
4.	LJSP Program Design Document (Volumes I and II)
5.	LJSP Contract – Scope of Services (as at latest amendment)
6.	LJSP Monitoring & Evaluation Framework
B.  LJSP Annual Program Plans
7.	LJSP Interim Program Plan 2003
8.	LJSP Annual Program Plan 2004
9.	LJSP Annual Program Plan 2005
10.	LJSP Annual Program Plan 2006 
11.	LJSP Annual Program Plan 2007
12.	LJSP Annual Program Plan 2008
D.	LJSP Reports
13.	Six month/Annual report (Apr-Dec 2003)
14.	Six month/Annual report (Jul-Dec 2004)
15.	Annual report (Jan-Dec 2005)
16.	Annual report (Jan Dec 2006)
17.	Annual report (Jan-Dec 2007)
18.	Annual report (Jan-June 2008)
19.	LJSP Activity Completion Report
D.  LJSP Contractor Performance Reviews
20.	Annual Contractor Performance Assessment 2003
21.	Annual Contractor Performance Assessment 2004
22.	Annual Contractor Performance Assessment 2005
23.	Annual Contractor Performance Assessment 2006
24.	Annual Contractor Performance Assessment 2007
25.	Annual Contractor Performance Assessment 2008
AusAID Response to LJSP Contractor Performance Reviews
26.	AusAID Response - Annual Contractor Performance Assessment 2003
27.	AusAID Response - Annual Contractor Performance Assessment 2004
28.	AusAID Response - Annual Contractor Performance Assessment 2005
29.	AusAID Response - Annual Contractor Performance Assessment 2006
30.	AusAID Response - Annual Contractor Performance Assessment 2007
31.	AusAID Response - Annual Contractor Performance Assessment 2008
Other AusAID LJSP Contractor Performance Reports
32.	Quality at Entry Report 2004
33.	LJSP Activity Monitoring Brief 2004
34.	LJSP Simplified Monitoring Toolbox 2005 
35.	LJSP Simplified Monitoring Toolbox 2006
36.	Quality at Implementation Report 2006
37.	Quality at Implementation Report 2007
38.	AusAID 1-page Contractor Performance Assessment Report 2003
39.	AusAID 1-page Contractor Performance Assessment Report 2004
40.	AusAID 1-page Contractor Performance Report 2005
41.	AusAID 1-page Contractor Performance Report 2006
42.	AusAID 1-page Contractor Performance Assessment Report 2007
43.	AusAID 1-page Contractor Performance Assessment Report 2008
E.  Justice Advisory Group Reviews and Analysis
Restorative Justice
44.	Application and Implications of GoPNG’s Law and Justice Policy and Plan of Action, as it relates to Restorative Justice - Assessment, April 2004*
45.	Working Paper on Restorative Justice and Community-Oriented Approaches to Crime Prevention and Dispute Resolution, July 2004*
Sector Facilities
46.	Law & Justice Sector Program Facilities Review and Strategy Development Mission – Review Report, January 2006 
47.	Sector Facilities Strategy Recommendations, January 2006 
48.	LJSP Facilities Technical and Financial Audit, November 2006
Community Justice and Crime Prevention
49.	Yumi Lukautim Mosbi Impact Evaluation 2006 Progress Report
50.	PNG Village Courts Support Review, April 2007
51.	Community Justice Liaison Unit Review, November 2007
52.	Annual Report 2005, Community Justice Liaison Unit
53.	Annual Report 2006, Community Justice Liaison Unit
54.	Annual Report 2007, Community Justice Liaison Unit
55.	Bougainville Review, Community Justice Liaison Unit, October 2008  
56.	Eastern Highlands Province Review, Community Justice Liaison Unit, October 2008  
57.	Bougainville Review, Community Justice Liaison Unit, October 2008 
58.	Supplement of Reports and Reviews, 24 December 2008 
Sector Budget Process
59.	Review of the 2008 Law and Justice Sector Development Budget Process, November 2007*
F. AusAID Reports and Commissioned Reviews & Analysis
60.	PNG Law & Justice Sector – Review and Contribution Analysis December 2006 (AusAID)
61.	AusAID Assistance to PNG’s Law and Justice Sector (2003-2007): Lessons Learned – Desk Review 2007 (Armytage)
62.	PNG Law & Justice Sector – Approaches to Capacity Building 2007 (Lyon)
63.	AusAID Assistance to PNG’s Law and Justice Sector (2003-2007): Monitoring & Evaluation 2007 (Kenway)
64.	AusAID assistance to PNG’s L&J Sector 2003 – 2007
65.	Rapid Review of Ombudsman Twinning Program 2008 (Lyon)
66.	AusAID PNG Program Gender Stocktake (2008)
67.	Office for Development Effectiveness report Violence Against Women in Melanesia and East Timor: A Review of International Lessons 2008
68.	AusAID Annual Sector Report 2006
69.	AusAID Annual Sector Report 2007
70.	Annual Review of Development Effectiveness 2007
71.	Gender Equality Annual Thematic Performance Report 2006-07
72.	Papua New Guinea Annual Program Performance Update 2006–07
73.	Assessment of the Indonesia Country Program Strategy 2003-2006, September 2007
G.  GoPNG Law & Justice Sector Policies, Strategies, Reports and Studies
74.	White Paper on Law and Justice, 2007
75.	National Law & Justice Policy & Plan of Action 2000 and Sector Strategic Framework 2005
76.	Law & Justice Sector Working Group, Law & Justice Sector Gender Strategy, 2005
77.	Law & Justice Sector Working Group, Provincial Engagement Framework, January 2006*
78.	PNG Law & Justice Sector Annual Performance Report 2004 
79.	PNG Law & Justice Sector Annual Performance Report 2005
80.	PNG Law & Justice Sector Annual Performance Report 2006
81.	PNG Law & Justice Sector Annual Performance Report 2007
82.	Success Stories in 2006 PNG Law and Justice Sector on the Move
H. Agency Documents
83.	Corporate Plan 2006 – 2010, Magisterial Service District Courts of PNG
84.	District Court Registry Enhancement Program: Number of cases registered in District Courts 2004 - 2008
85.	Corporate Plan 2009 – 2011, Office of the Public Solicitor, PNG
86.	Annual Report 2008, Office of the Public Solicitor, PNG
I.  Additional Documents
87.	Draft Baseline Collation Report
88.	PNG JAG ICR Final Draft Report
89.	JAG Final Activity Completion Report
90.	Final LJSP CPA Report
91.	Draft Provincial Engagement Research and Evaluation Report (July 2009)
92.	PNG Law and Justice Sector Provincial Engagement Framework Final Draft, 26 January 2006
93.	Provincial Engagement, S. Mokis, 7 August 2006
94.	Provincial Engagement – Powerpoint Presentation to Provincial Administrators Consultative Meeting, Gateway Hotel 15-16 May 2008
95.	Integration of Cross Cutting Issues into Law & Justice Sector Activities Approaches adopted by the Facilities & Assets Support Team
96.	An Experimental Approach to Monitoring Capacity and Capacity Development: The Case of the Magisterial Service, Draft of Report Covering 3 Phases, Heather Baser, January 2009
97.	JAG Review of the 2007 Development Budget Process, 20/11/06
98.	JAG Review of the 2008 Development Budget Process, 2/11/07
99.	Lessons Learned in First Phase of the Law and Justice Sector Program From 2003 – 2008 Involving 8 National Agencies which Administer the Criminal Justice System in PNG Being the RPNGC, NJSS, MS, CS, DJAG, PP, PS and OC. Clivson Philip, Registrar of District Courts (PNG) and member of LJSWG, 2/8/09.

J.   AusAID Templates

100.	Template for Independent Completion Report
101.	Template for Aide Memoire for Evaluation 
102.	Guideline: Manage the Independent Evaluation of an Aid Activity (effective 12 November 2008 – 30 November 2009)
103.	AusAID Standard Evaluation Questions (template current to 30 November 2009)

Annex 4: Itinerary and persons consulted

Date	Time	Agenda	Venue/Status	
Mon 20 July	8.30 – 10.00	Gabriel Kubul, Senior Program Officer, Jason Kalimba, Program Support Officer, Law & Justice Unit, AusAID Post; Joanna Houghton, Manager, PNG Branch/Economic & Public Sector Governance, AusAID, Canberra; Dr Penelope Murphy, ICR Team Leader.  	AHC	
Tues 21 July	8.30 – 10.00	John Dinsdale, AusAID Law & Justice Adviser, located with LJS Secretariat (former Program Manager, LJSP 2005 – 2008); Katherine West, 1st Secretary Law & Justice, Gabriel Kubul, Ms. Tau  Hoire, Law & Justice Unit, AusAID Post; Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow, ICR Law & Justice Specialist.	AHC	
	10.00 – 10.45 	Peter Pascoe, Strategy & Change Coordination Adviser, PALJP (former LLJSP RPNGC Strategy & Coordination Adviser; Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
	11.00 – 12.30	Mick McIntyre, Court Registry Adviser PALJP (former LJSP Court Registry Adviser); Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
	2.00 – 3.30	David Ward, Adviser Ombudsman Commission, PALJP (former Adviser/Twin, Ombudsman Twinning Program); Joanna Houghton, DrPenelope Murphy, Dr Heike Gramckow.  	PALJP Office
Old Yacht Club	
Wed 22 July	9.00 – 10.15	John Rennie, PALJP Deputy Team Leader, Agency based Advisers (ex. Justice Advisory Group); Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office
Old Yacht Club	
	10.30 – 11.45	ICR team meeting: Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.   	Crowne Plaza Hotel	
	1.00 – 1.45	Dr Rodney Kameata, Community Justice Liaison Unit; Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	Department of Community Development, Waigani	
	2.40 – 3.20	Clivson Philip, Registrar, Leonard Mesmin, Deputy Registrar, Magisterial Services; Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	Magisterial Services Office, Port Moresby	

Date	Time	Agenda	Venue/Status	
	3.30 – 4.45	Bob Shillabeer, Facilities Development Practitioner, PALJP (former Regional Facilities Adviser, LJSP), Brian Boon, Assets Management Development Practitioner, PALJP (former Assets Management Adviser LJSP); Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office 
Old Yacht Club	
Thurs 23 July	Public Holiday	ICR Team Planning Discussion/clarification of TORs: Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	Crowne Plaza Hotel	
Fri 24 July	9.00 – 10.15	Kepas Paun, (Former Director of LJS Secretariat), Adviser Restorative Justice, LJSP; Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office 
Old Yacht Club	
	10.15 -11.00	Stephen Mokis, Adviser Provincial Engagement, LJSP; Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office 
Old Yacht Club	
	1.00 – 2.30	Chronox Manex, Chief Ombudsman; Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	Deloitte Tower Office of Ombudsman Commission	
	3.00 – 4.30	Jane Kesno, Gender Adviser LJSP and PALJP & Lead Adviser Cross-Cutting Issues; Ghang Oyang, Adviser HIV/AIDS, LJSP and PALJP; Joanna Houghton, Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office 
Old Yacht Club	
Sat 25 July	11.00 – 12.30	Discussion of draft outline of IC Report. 
Dr Penelope Murphy, Dr Heike Gramckow.  	Crowne Plaza Hotel	
Sun 26 July	11.00 – 12.00	Revision of draft outline of IC Report. Dr Penelope Murphy, Dr Heike Gramckow.  	Crowne Plaza Hotel	
	5.30 – 6.00	Review and discussion of draft outline of IC Report with Joanna Houghton; Dr Penelope Murphy, Dr Heike Gramckow.  		
Mon 27 July	8.30 – 10.00	Bruce Grant, Chief, Child Protection; Anthony Nolan, Child Protection Specialist, UNICEF; Dr Penelope Murphy, Dr Heike Gramckow.  	UNICEF Office 
Deloitte Tower	
	11.00 – 12.00	Kirsten Bishop, Law & Justice Adviser, AusAID, Canberra (former 1st Secretary, Law and Justice, AusAID Post).  Teleconference.  Joanna Houghton; Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	

Date	Time	Agenda	Venue/Status	
	4.00 – 5.00	Steve Sims, District & Provincial Coordinator (formerly of LJSP, currently of PALJP); Joanna Houghton; Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office
Old Yacht Club	
Tues 28 July	8.30 – 10.00	Harman Buago, First Assistant Secretary, Policy and Planning; Negil Kauvu, Director, Community Based Corrections and Juvenile Justice, Department of Justice and Attorney General	Sir Buri Kidu Haus
Waigani	
	10.15 – 12.00	Ken Richardson, Adviser, PALJP (former Information Management & Technology Adviser, LJSP). Joanna Houghton; Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office
Old Yacht Club	
	1.00 – 2.30	Inspector Joanne Clarkson, Royal PNG Constabulary.  Joanna Houghton; Dr Penelope Murphy, Dr Heike Gramckow.  	RPNGC Headquarters, Konedobu	
Wed 29 July	9.00 – 10.15	Colin Adams, Program Director PALJP (Former Program Director, LJSP). Joanna Houghton; Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
	1.30 – 3.15	John Toguata,  Lou Grima, Integrity Systems & Anticorruption Development Practitioners, PALJP (formerly Integrity Systems & Anticorruption Adviser, LJSP); Joanna Houghton; Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office 
Old Yacht Club	
	3.15 – 4.30 	Stanley Raka, Senior M&E Adviser LJSP and PALJP; Joanna Houghton; Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office
Old Yacht Club	
Thur 30 July	9.30  - 	Ms. Tau  Hoire, Law & Justice Unit, AusAID Post; Rebecca Robinson (former Deputy Team Leader, LJSP), Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
	11.00 – 12.00	Rebecca Robinson, (former Deputy Team Leader, LJSP) Teleconference. Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
	1.00 – 2.30	Joanne Choe, Program Director, AusAID Sub National Strategy, AusAID Post (former Activity Manager, LJSP) Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	

Date	Time	Agenda	Venue/Status	
	3.00 – 4.30	Iva Kola (former Manager Implementation and Acting Director LJS Secretariat). Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
Fri 31 July	8.30 – 10.00	Helen Disney  Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP  Office
Old Yacht Club	
	10.15 – 12.00	Alois Francis, Deputy Executive Officer, CIMC. Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	5th Floor Mogoru Moto Building	
	1.30 – 2.30	Rose Koyama.  Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
Sat 1 August	4.00 – 5.15 	Bridget Laimo, Contractor Performance Assessment Finance & Audit Specialist. Dr Penelope Murphy.  	Crowne Plaza Hotel	
Sun 2 August	10.00 – 12.00	Team rating of criteria. Dr Penelope Murphy, Dr Heike Gramckow.  	Crowne Plaza Hotel	
	4.30 – 5.30 p.m.	Drafting of Aide Memoire. Dr Penelope Murphy, Dr Heike Gramckow.  	Crowne Plaza Hotel	
Mon 3 August	10.15 – 11.00	Benny Daniel, Acting Corporate Services Director, Public Solicitor’s Office.
Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	Public Solicitor’s Office
2nd Floor, Garden City, Boroko	
	1.30 – 2.45	Vicki Morris, HR Adviser, LJSP. Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
	2.45 – 4.15	Josephine Gena, Community Development Adviser, LJSP. Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
Tues 4 August	10.30 – 12.00	John Dinsdale, AusAID Law & Justice Adviser, located with LJS Secretariat (former Program Manager, LJSP 2005 – 2008). Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC	
Wed 5 August	10.15 – 12.00	John Mooney, former SPED, LJSP. Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	Teleconference
AHC	
Date	Time	Agenda	Venue/Status	
	2.00 – 3.00	Colin Adams, Program Director PALJP (Former Program Director, LJSP). Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office
Old Yacht Club	
	3.00 – 4.45	Joe Kanekane, Director LJSS. Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	PALJP Office
Old Yacht Club	
Thur 6 August	2.30 – 4.00	Presentation of Aide Memoire to representatives of DNPM, the Sector, AusAID and the Contractor. Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC Conference Room	
	4.00 – 5.00	AusAID Law and Justice Team – Final Debrief. Joanna Houghton; Clivson Philip, Dr Penelope Murphy, Dr Heike Gramckow.  	AHC Conference Room	

Annex 5: Compliance with the Paris Declaration on Aid Effectiveness and Accra Agenda for Action
This attachment contains a summary analysis of the extent to which LJSP complies with the Paris Declaration on Aid Effectiveness of March 2005 (hereafter “Paris Declaration”) and the Accra Agenda for Action of September 2008 (hereafter “Accra Agenda”) as called for in the template for the ICR.  The PNG Commitment on Aid Effectiveness of July 2008 (hereafter “PNG commitment”) pre-dates the Accra Agenda and is framed around the five partnership commitments of the Paris Declaration: Ownership, Alignment, Harmonisation, Managing for Results, and Mutual Accountability.  The Accra Agenda aims to “accelerate and deepen implementation of the Paris Declaration.” It focuses on three major challenges: Strengthening Country Ownership over Development; Building More Effective and Inclusive Partnerships for Development; and Delivering and Accounting for Development Results. All three documents are mutually consistent. However the PNG Commitment, being specific to PNG, is the one most directly relevant to the LJSP and is therefore the one to which particular reference will be made in this summary.  All three documents relate to collective behaviour at the country level. Only some aspects are directly applicable to the program level.   The approach taken therefore in this analysis is to discuss the performance of LJSP loosely under each of the five shared principles of the PNG Commitment which correspond with the five partnership commitments of the Paris Declaration.
Ownership
Perhaps more than any other AusAID program in PNG, the LJSP has enjoyed clear PNG government ownership.  This is because it directly supports an initiative that came out of the PNG Department of National Planning and Monitoring (DNPM) in 1997 to confirm and consolidate common issues as a basis for formulating PNG’s National Law and Justice Policy and Plan. The resulting Policy and Plan of Action was adopted and endorsed by the National Executive Council (NEC) in August 2000 and the LJSP was designed expressly to support its implementation.  In 2007 the NEC endorsed a White Paper on Law and Justice which endorsed the policy and the Sector Strategic Framework (SSF). Ownership continues to be assured by the existence and role of the National Coordinating Mechanism which, with support from its Law and Justice Sector Working Group and Secretariat, governs the work of the sector as a whole.  This makes the LJSP a stand-out program in terms of partner government ownership, subject to the following three main limitations. 
From the PNG Government side, some clarity has been lost during the course of the program regarding the institutional home of the NCM, the Law and Justice Sector Working Group and the Law and Justice Sector Secretariat within the PNG government organisational structure – a matter needing resolution by the partner government (see Recommendation 13). 
During the LJSP’s design phase in 2003, non-government organisations (NGOs) and faith-based organisations were part of wide consultations. However, since then there has been very limited participation of such agencies in defining the Sector’s operational development policies. This has tended to reduce the potential for aid effectiveness.
The Accra Agenda states that “Developing countries will systematically identify areas where there is a need to strengthen the capacity to perform and deliver services at all levels – national, sub-national, sectoral, and thematic – and design strategies to address them.” Although the sector, with support from the program, has gone part way to identifying such areas, much further work remains to be done.  The scope for a future program to strengthen the capacity to deliver services, especially at sub-national level, and to support the sector in designing strategies to address them, remains very great.
Alignment
LJSP clearly represents a program-based approach with flexible implementation mechanisms, and was clearly linked to the Medium Term Development Strategy and the National Law and Justice Policy and Plan. Its primary focus was on strengthening institutional capacity within the sector, and thus contributed to Public Sector Reform.  Operationally, very close alignment of the LJSP with the PNG government development budgeting system was achieved.  All program support had to be obtained via GoPNG Project Formulation Documents prepared and submitted to the NCM through the Law and Justice Sector Working Group and the Law and Justice Sector Secretariat by the agencies or Activity Management Teams within the sector.  This included all of the technical advisory and other support supplied by the program.  The process of channelling program support through this path emerged at an early stage of the program. 
The following main limitation applies:  Both GoPNG and AusAID formally acknowledged, through program design documentation, the importance of non-state-actors as key partners in achieving effective aid outcomes through LJSP.  However the strong focus on using the government budgeting system has not been matched by a strong involvement of non-state-actors in the development and implementation of the program’s development priorities. Indeed, it seems possible that there is an inverse relationship between strengthening government systems and non-state engagement, i.e. that as government systems are strengthened, the more non-state actors are excluded – unless conscious measures are taken to prevent this from happening. 
The Accra Agenda clearly accepts that donors may “choose to use another option and rely on aid delivery mechanisms outside country systems,” but if so should “transparently state the rationale for this and review their positions at regular intervals,” establishing “additional safeguards and measures in ways that strengthen rather than undermine country systems and procedures.” It also commits donors and partner countries to deepen their “engagement with civil society organisations (CSO) as independent development actors in their own right whose efforts complement those of governments and the private sector…”  As part of that process, donors and partner countries committed to seek to “i) improve co-ordination of CSO efforts with government programmes, ii) enhance CSO accountability for results, and iii) improve information on CSO activities.” 
The National Law and Justice Policy and Plan of Action clearly recognises that Government alone cannot tackle all the problems, that communities must also accept responsibility and play their parts, and that there are serious deficiencies with the classic model of criminal justice. The vast majority of PNG’s people live in rural areas and are beyond the reach of the formal law and justice system, and this is likely to remain so for the foreseeable future.  The Sector and its agencies have experienced difficulty in engaging rural communities through civil society organisations in planning ways to improve restorative justice approaches and in ensuring that these are budgeted and prioritised through the Development Budget process, despite being prompted to do so. 
A clear case can be made therefore that, in the spirit of the Accra Agenda, in a subsequent program AusAID should retain and, as appropriate, exercise an option to provide dedicated support for community and civil society engagement in the sector.  This need not require the establishment of a parallel funding system.  Approaches might include, for example, the provision of awareness to sensitise agencies as a cross cutting issue to their obligations to implement these aspects of the national policy; additional resources to the cross cutting team to carry out this awareness raising/capacity enhancement; provision of incentives to agencies to engage with communities and civil society organisations; the use of selection criteria applied to that subset of project formulation documents that are to be funded by AusAID through PALJP and so on. The creation of the CJLU may have contributed to the limited attention to strategic and ongoing community and NGO engagement with the sector and its agencies; focusing this function on the CJLU instead making it a shared responsibility that over time evolves and is institutionalised.
Harmonisation
Harmonisation of donors in the PNG Law and Justice Sector is less complex than in some other sectors as a result of the fact that AusAID is by far the largest donor to the sector.  There is scope for improving communication between AusAID and UNICEF, however. Missions have been held jointly with GoPNG and analytical work has been shared, although GoPNG stakeholders appear increasingly to be experiencing fatigue with participation in joint missions or to lack the capacity to participate fully. This is evidenced by the limited GoPNG participation in this ICR mission.  GoPNG opportunity to benefit from involvement has therefore been limited. 
AusAID appears to have maximised its delegation of authority for LJSP to its Post in PNG and there has been continuity of AusAID staff involved with monitoring the program. However AusAID’s attempts to rely on GoPNG and Program M&E systems have proven frustrating and much less than satisfactory (See Section 2.7 for further discussion.)
Managing for Results
Managing for results has been severely limited by an incoherent approach to M&E in the  program and the sector, and a failure by the program to adequately identify and address capacity constraints in generating and using management information, and in reporting on results. This would inevitably impact on the overall ability of GoPNG and AusAID to manage resources and decision making for results (See Section 2.7 for further details).
The Accra Agenda commits donors to improve management for results by aligning their monitoring with country information systems, and to support and invest in strengthening developing countries’ national statistical capacity and information systems. Within the law and justice sector there is much to be done to improve the statistical capacity and information systems before reliance can be placed by AusAID and GoPNG on these for management for results.  Continuing support for capacity improvement is needed, and until this is substantially achieved there is a clear case for AusAID to undertake independent collections baseline and periodic collections of data. 
Mutual Accountability
Assessment of GoPNG’s and AusAID’s ability to account for aid effectiveness is beyond the scope of this ICR. It may be assumed however that weakness of the Program in its ability to report on results also impacted on the ability of the key stakeholders to account for aid effectiveness with respect to LJSP.
Summary and Conclusion
LJSP has substantially complied with a number of the broad intents of these guiding documents, in particular those relating to ownership and alignment.  LJSP has been consistent with the intent of harmonisation, although the need for harmonisation has been much less in this sector than others, as AusAID is the only main donor to the sector. There is less clear evidence of compliance with Managing for Results.  The applicability at the program level of mutual accountability is also less clear. 

Annex 6: The effectiveness of LJSP in supporting the five goals of the Sector Strategic Framework
This Annex contains a review of the effectiveness of the LJSP in meeting its objectives against the Outcomes and Objectives listed in Attachment D to the TOR. As Attachment D does not identify Objectives as such, the objectives are understood to be those implied by the five goals of the Sector Strategic Framework. These are: Improved Policing, Safety and Crime Prevention; Improved Access to Justice and Just Results; Improved Reconciliation, Reintegration and Deterrence; Improved Accountability and Reduced Corruption; and Improved Ability to Provide Law and Justice Services.  This section is structured around those goals.
Goal 1: Improved Policing, Safety and Crime Prevention:  The main focus had been on developing the corporate capacities of the national agencies and strengthening some crime prevention efforts. Good strides had been made there. Regarding support for the Royal PNG Constabulary (RPNGC), “almost 80% of the recommendations from the Police Administrative Review (2004) have been implemented, resulting in improved operational and administrative practices and greater police accountability and discipline.”  Without a more detailed review of police operations it is difficult to verify this statement or to gauge the effectiveness of these changes. Some important improvement processes related to financial management had taken firm hold. Others, such as a review of standard operating procedures, had only recently been undertaken. Views were expressed that human capacities to carry on and expand improved corporate activities were still limited.  A relatively small number of staff working in human resources (HR) and budget and finance had been trained in applying new processes and using IT systems, although staff turnover was high.  In early years advisers typically engaged with partners one-on-one, with little use of other capacity building concepts. This limited the effectiveness of program input especially in HR operations and likely in other areas. A significant lack is the creation of a pool of local trainers (from within the police and from other agencies) to serve as multipliers.  
Improved service delivery through improved frontline policing was not a core focus of the program.  As a result, earlier improvement efforts in this area experienced a setback when the policing component of the Enhanced Cooperation Program was withdrawn in May 2005. Due to the position taken by GoA that no advisers could be provided to support frontline police functions, the program had few options to redirect funding to address this significant gap.  However, the inclusion of the RPNGC in the sector provided a number of opportunities to continue to engage and assist in strengthening core police functions through other programmatic areas. These include specialised family and sexual violence projects, anti-corruption and anti-fraud projects, local crime prevention efforts, and juvenile justice projects. The latter were spearheaded largely by UNICEF.  Neither the 2007 White Paper on Law and Justice in PNG, nor the following Annual Program Plans indicate a particular attention to strengthening frontline police functions beyond those efforts.  Similarly, program expenditure for police development activities was only K15,911 for the duration of the program. This is low, when compared to other agencies and the fact that police was by far the largest justice sector agency.  Accordingly, it is not surprising to find that program support to improve policing had not been – and probably could not be - very effective and that police performance remained problematic. 
There were a number of positive developments in community based crime prevention. In the NCD, the Yumi Lukautim Mosbi (YLM) project, which started outside of the program, became a focal point for LJSP assistance only in 2007. It was aimed at developing and implementing crime reduction and urban safety initiatives targeted at grass roots communities. This very inclusive effort built upon community engagement. It brought all available resources together, and addressed the needs of women and youth. It reflected key problem solving and restorative justice aspects, and is an effective crime prevention effort. Effectiveness of other public safety and crime prevention efforts in the Eastern Highlands Province and Bougainville could not be directly reviewed by the ICR. Progress in provincial engagement review was limited. The effort involved in developing the necessary relations in the initially selected 9 provinces was underestimated. Limited capacities to plan for such significant engagement and the logistics support required for ongoing coordination and engagement continued to hamper effective engagement over time.  
Goal 2: Improved Access to Justice and Just Results: Limited infrastructure exists to connect rural communities. Cultural practices vary widely and terrain is difficult. The provision of adequate access to justice in a manner that is responsive to the different needs of communities across PNG is a significant challenge. The creation of village courts was an important response. Although village courts still had a long way to develop, they were starting to take on an increasing role in providing dispute resolution alternatives at the village level, and were becoming more accessible to women. Program support for the successful expansion of improved registry services and the creation of two model courts have also effectively enhanced access to justice in those locations. Especially remarkable is the expansion of legal aid provision to the current 8 provinces plus Bougainville, and the creation of legal aid desks in 3 additional locations. Considerable awareness and outreach activities had been supported by the program through the Community Justice Liaison Unit (CJLU), the Office of the Public Solicitor (PS), and other activities such as YLM. These increased access to justice through a better understanding of the justice sector function and how to access it. It was not clear, however, whether program support for these efforts had been effective in increasing understanding by different sectors of the public, especially those outside of Port Moresby.  
Significant changes could be observed in sector responses to juvenile justice. Eleven new Juvenile Courts and four new juvenile reception centres were created. A new National Juvenile Justice Policy was now in place as were a Juvenile Court Protocol for Magistrates, a National Police Juvenile Policy and Protocols and Minimum Standards for Juvenile Institutions.  Since UNICEF had been a driving force in advancing access to justice and more just results for juveniles it is difficult to establish the program’s exact contribution to these outcomes. The combined effort has been very effective and underlines the importance of linking the sector and program activities to other efforts and resources available.  
Goal 3: Improved Reconciliation, Reintegration and Deterrence: Despite the inclusion of restorative justice in the National Policy and the SSF, program support for improved reconciliation and reintegration only gained momentum in the last year. This was largely as a result of linking to other ongoing efforts, such as YLM. The increased efforts to enhance village court operations and the creation of viable community based correction options with a strong focus on enabling offenders to gain employment indicate that the program had become more effective in connecting to the needed community resources to create these alternatives. The Restorative Justice AMT was only revitalised, however, in 2008, not soon enough to be effective during the first phase of the LJSP. It is difficult to establish whether restorative justice and other program areas were neglected because the sector lacked capacities to lead or react, or because the focus of the program was on other areas. Little information is captured in program documents that would indicate what efforts had been made to move this key goal forward. Most justice sector agencies around the world also have difficulty in defining restorative justice in terms of core justice sector agency operations. The lack of program support to further the discussion and development of an operational model of restorative justice for PNG is surprising. The efforts towards the end of the program suggest that this challenge may be tackled in future through the PALJP.  
Goal 4: Improved Accountability and Reduced Corruption: The program was quite effective in supporting the development of agency systems that increase accountability, such as improved budgeting and financial management capacities, internal review and reporting systems across all justice sector agencies. The establishment of annual reporting by all key justice sector agencies was an important milestone for increased accountability of justice sector agencies.  Interestingly, a sector report that would mirror the good agency reports is not being published.
Support for improved operations in several key accountability agencies had been quite effective, especially the Ombudsman Office and Commission, and the National Anti Corruption Agency. Support to the Solicitor General’s Office was provided only towards the end of the program in 2008. It created improved capacities to track and successfully pursue claims the State against companies and individuals. These often involve significant sums owed in licensing fees or taxes.  Furthermore, support to the creation of the Financial Intelligence Unit within the RPNGC had effectively created capacities to successfully pursue corruption and fraud cases. Also, integrity reviews had been conducted for several key agencies. While it is difficult to distinguish program support for the latter from sector activities, it is unlikely that such integrity reviews would have been conducted in such a short time – if at all – without the assistance of the program.
Goal 5: Improved Ability to Provide Law and Justice Services:  The program focus on creating well performing “corporate systems” mainly meant creating budget and finance capacities, basic HR functions and IT systems to support those core corporate functions.  The emphasis on creating corporate capacities was based on the argument that solid performance and control over finances and human resources are at the core of well functioning agencies. Whether these improvements manifest themselves in improved capacities to deliver justice sector services remains to be seen. Although key for any organisation, these functions have only limited impact on the agencies’ core frontline operations. Good management, supervisory capacities, and change management are important. But needs and results based planning, relevant training capacities, and frontline staff training still remain to be addressed. There is little indication that either had been a strategic focus of program activities, but rather seems to have been left to the initiative of individual advisers.
Improved ability to provide law and justice services, and related program effectiveness differed across the multiple agencies, with good results especially in the Ombudsman Commission, the Magisterial Services, and the Public Solicitor (and possibly the Solicitor General (SG)).  Capacity building support to the OC, the MS, and SG extended very much to creating broader management capacities as well as capacities to deliver core services – and their ability to fulfill their mandates as required by local and international law.  For other agencies, especially the police, this remained a great challenge.  
Overall, although effectiveness across the five program goals was mixed, the program was very successful in facilitating the development coordination structures and translating the meaning of a sector wide approach to agency planning, operations, management and policy.  

Annex 7:  Risks to achievement of program objectives
A number of risks to achieving the objectives of the program are discussed below.  
Risks to achievement of objectives: Risks are posed by changes in agencies’ leadership that impact commitment to the program’s goals, changes in government support, and shift in recurrent budget levels and priorities.  Lack of control over these key risks does not mean lack of risk management strategies to predict, plan for and prepare for such changes.  This includes continuous engagement with key decision makers and attention to future developments – as unpredictable as they may be.  In a sector wide approach the key sector stakeholders need to manage this risk. This requires a dedicated focus on creating their capacity to do so. It was not apparent to the ICR Team whether the program adequately captured information on impending PNG leadership or policy changes to feed into program adjustments or whether key counterparts were actively engaged in an ongoing assessment of such risks.  
Key PNG agency informants recognised and mentioned that there had been a lack of information and “lobbying” of the government. The limited communication about the program’s effort to the government other than the standard reports submitted seems to be an area that was in need of improvement.  This was seen as an important task to be tackled.  Government had not maintained anticipated funding to the sector. At the same time significant sums had been dedicated to justice sector improvements at the local level as a result of District Improvement Support Program grants via the Open Members of parliament. This new policy dedicated to each district K1 million for justice sector reform– money that was not yet tied to the program or the national strategy and was outside of the sector’s control. There was no strategy, structure or activity in the LJSP to respond to this very important change. Moreover, there did not appear to be one in the design for the PALJP, even though this had been proposed two years previously.
Another important risk was lack of grounding in local community level realities. There was strong focus on national level justice sector agencies, limited input and engagement of civil society, and very limited emphasis on gathering information and data especially at the community level. This severely impeded the program’s and sector’s ability to expand its reach to the provincial and district level in a strategic and informed manner. The pilot activities in Port Moresby and two provinces provide some examples of engagement but little information about the differing needs and capacity variations across all provinces. Furthermore, only a few civil society organisations were currently connected to the sector formally. The role and importance of the CJLU had not really been explored. Despite the very unique and strong focus on restorative justice of the PNG justice sector policy and strategy, which requires significant civil society involvement, the program had provided limited to support to ensuring this vital connection was made. Civil society leaders interviewed expressed lack of interest in actively engaging because sector meetings were too much focused on bureaucratic issues instead of substance.  Similarly some important international NGOs such as UNICEF, while supporting projects that were part of efforts important to the sector, were not appropriately linked to the sector nor regularly informed about key program activities beyond the direct engagement project. 
Only towards the very end of the program did the contractor finally bring in a qualified local consultant who understood the need for a more strategic engagement of key NGOs operating in PNG and the engagement processes needed.  This was a step in the right direction that should greatly contribute to program progress under PALJP.
One serious problem area throughout the program duration that was not resolved was the lack of baseline information and inefficient M&E capacities and systems.  This issue is addressed in more detail in Section 2.7.  It should, however, be noted the lack of sufficient M&E information further impeded early recognition of problem areas.  Throughout much of the program period there was inadequate contractor response to continuous requests from AusAID for improved M&E.
Effectiveness of program support to the police was especially low. From the beginning, support remained largely focused on developing the corporate capacities of the police as well as infrastructure improvements.  The program design seemed to have assumed that the Australian Federal Police would have been available to provide significant support for improving core police operations and service delivery. The withdrawal of this assistance therefore left a significant gap.  The focus on corporate functions was supported by three advisers only – not much for such a large organisation.  Over the years, good adjustments in key fiscal management operations were made, HR policies were greatly improved which resulted, among others, in an increase in female recruits.  However, the capacity building approach had little focus on broadening capacities beyond those directly engaged with the international adviser.  After more than six years of program support no pool of local staff had been created that could continue the training of new staff in HR functions.  
Another important corporate capacity area that still needed to be handed over to the agencies after six years of assistance was procurement for construction and asset maintenance.  This is not a satisfactory outcome.  Interestingly, however, infrastructure improvements in police housing had been used very creatively to teach basic day-to-day maintenance of housing units and as an entry point for creating awareness about key cross cutting issues, such as family violence and HIV.
Overall, it remained to be seen if the focus on creating sector coordination mechanism and strong corporate capacities enabled the sector agencies to expand and improve their service delivery in a sustainable manner – which is the main goal of the national policy and strategy.  Reviewing the progress of the program in various activity areas it is obvious that much hinged on finding the right advisers.  The strongly expressed support and continued need for select advisers with a good track record in individual agencies is an indicator of how effective individual program staff members were.  The more recent move of the program to increase the number of local advisers was a very positive step. Several of them had achieved significant success in relatively short periods of time – very good value for money.

Annex 8: Impacts from external factors
A number of important external events impacted on program performance since inception. These are summarised below. The program’s impact was limited by its inability to address these adequately. While these events were largely outside the control of the program or the sector, they still needed to be managed. As outlined below, there is little documentation or other evidence of strategic responses to such significant events.
Probably the most critical external factor was the difficult situation that developed when the Australian Government withdrew most of the support to the RPNGC provided under the Enhanced Cooperation Program. This meant that a significant number of technical advisers were not available to support core frontline and operational activities. It also meant that the program was unable to field its advisers for similar activities.  This created a significant gap which likely resulted in a deterioration of earlier improvements in line services provided by the RPNGC.  Despite this the program was able to continue its support for strengthening core corporate capacities.  This incident demonstrated the importance of being part of the sector for each justice sector agency. Being part of the sector meant support, local and international, could continue even if it was more limited.  It is unclear, however, whether the program recognised and adjusted to this challenge appropriately.
Another perhaps less negative but important external factor in the long run was the continuous misunderstanding of the role and responsibility of the DNPM related to the sector.  The important function of linking the sector to overall government policies, funding and operations rested with the DNPM.  Its monitoring branch was required to support the sector to achieve a sector monitoring framework for service delivery in the districts. The DNPM’s decreased engagement with the sector was a significant hurdle in many respects, especially in ensuring adequate communications with the government on budget issues and for monitoring functions.
A range of GoPNG wide policy efforts and other broader development assistance programs had impacted (and would continue to impact) justice sector and program efforts.  Two of the more important and recent ones are described below. None of these important initiatives had been adequately addressed by the program so far, if they have been addressed at all.  
•	The allocation in 2008 and 2009 of K5 million to each Open Member of parliament through the District Support Improvement Program for expenditure in each of the 89 districts. K1 million of the funding was allocated to law and order projects.  This funding has been allocated without any coordination with the sector, which has yet to establish law and justice strategies for all provinces through its PEF.
•	The GoPNG conducted a development forum in September 2008 to gauge views of academia, bureaucrats and civil society to develop a 40 years development strategy of 2010 to 2050. The plan will be announced by the Prime Minister on the anniversary for the country’s Independence Day on Wednesday 16th September 2009. The law and justice sector had been given only two weeks to have its component of the sector development strategy submitted to the working committee for incorporation into the document. 
•	The Provincial Performance Improvement Initiative (PPII) began on a pilot basis in mid-2004. It focuses on strengthening the capacity of 18 provincial administrations in PNG to better deliver effective services (health, education, infrastructure, and law and order) to the people. This long-term capacity development program has a time frame of 15 to 20 years, In 2008 PPII worked in 13 of the provinces of PNG. The program is led by the Department of Provincial and Local Level Government (DPLGA) and is also supported AusAID, The PPII targeted pilot provinces together with AusAID’s SNS program have commenced mapping out how best districts can be developed not only through law and justice but also in other sector priority areas now that the PNG government have prioritised service delivery at the 89 districts.  
On the positive side, there had been increasing interest from civil society and business in supporting selected sector initiatives, such as YLM. Individual justice agencies and development advisers recognised the potential of the linkages but there was insufficient strategic program support and reaction to these opportunities.  Many more linkages needed to be identified and established to ensure that the program and the sector were effectively making use of these additional opportunities.  The forthcoming World Bank funding for a youth employment program was apparently triggered by the successes of YLM and will be another important opportunity to establish systematic links.

Annex 9: Chronological Summary of Monitoring and Evaluation in LJSP
The LJSP contract stipulated that the Program Design Document to be produced by the contractor should include a monitoring and evaluation Framework (MEF) for the program and baseline data against its key performance indicators (KPIs), informed by and consistent with the work of the Justice Advisory Group (JAG).  The resulting LJSP Evaluation Framework (Sept 2004) reflected the Performance Monitoring Framework (PMF) of the JAG, and its 61 indicators.  By December 2004 it was decided that the “MEF must directly serve LJSP rather than LJSP being expected to serve an overly elaborate M&E system.” In February 2005 the contractor hired two M&E consultants to further develop the Program MEF and M&E Plan and a draft framework was submitted to AusAID in March 2005.  The agreed 2005 format was used to prepare the 2006 MEF. A draft was submitted to AusAID in March 2006 with the final version agreed in July 2006. By this stage the program had been in implementation for two and a half years, but no baseline data had apparently been collected. 
LJSP’s 2006 Annual Program Plan (APP) and 2006 MEF are reported to have attempted to tighten the description of LJSP’s contribution to the sector’s outputs and outcomes, and describes what were considered to be significant improvements in the 2007 APP. This was seen as the start of LJSP’s evaluation of its contribution to the development of the sector’s and agencies’ capacity to design, implement, monitor and report on activities that implement the SSF and the NLJ Policy. From this point on Annual Reports presented useful summary evaluation assessments of the program’s contribution in terms of relevance, effectiveness, efficiency, impact and sustainability in each of 12 key result areas: local crime prevention; protection of children and women; processing serious crime; court processes; restorative justice; juveniles and rehabilitation; corrections; fraud and corruption; improved public administration; and civil society engagement. In 2007 restorative justice was subsumed into another key result area, and sector coordination, HIV and AIDS and Gender were added. The 2008 Annual Report departed from this reporting format. 
The 2007 Annual Plan contains a description of a “Sector Performance Monitoring Framework” explaining that from the SSF and with the assistance of the JAG, it had been possible to identify a small set of critical measures or expected outcomes (key performance measures) that could be used to monitor success and to assist stakeholders in effective management.  Responsibility for data collection was stated to be part of the core business of public servants.  
It was not until 2007 that it was acknowledged that in order for LJSP to fulfil its obligations to AusAID and the sector to account for its contribution there needed to be a link from the LJSP level of M&E into agency, and some sector M&E. Further, that LJSP outputs are only part of a broader set of PFD outputs that collectively contribute to improvements, and need to be read as such. A significant achievement in 2007 was considered to be that the advisers’ work was now clearly located within the sector’s activities. The flow from the sector’s M&E framework, through the agency annual plan, to the PFD/DB and finally into LJSP’s contribution was seen as much clearer. At this stage LJSP accepted that its own M&E, at the programmatic level, required it to work with the agencies to advance sector M&E capacity for Development Budget (DB) activities, Project Formulation Documents (PFDs) and agency annual plans. AusAID and the sector agreed to support the agencies to improve M&E through the provision of an additional Agency Performance Monitoring Adviser who commenced with LJSP on 1 October 2007.
An MEF was produced in January 2008 for the remaining 15 months of the program (i.e. up to April 2009), updating the 2007 format and providing more detail and context on M&E activities at the sector and agency levels. Revisions were aimed at significantly enhancing program support for building agency organisational capacity to establish and maintain robust M&E and reporting systems; and at strengthening program M&E and reporting on the impact of the Australian contribution to the sector and agency activities funded through the DB. The MEF contained a new Logframe (Annex 1) intended as a higher level reporting tool for the LJSP MEF primarily to help capture outcomes and impact from those DB activities supported by AusAID. These “(plus gender) bring together a number of otherwise discrete activities in a manner consistent with particular SSF strategies where significant Australian support is provided.” 
This MEF was a break-through in terms of clarifying thinking.  It clearly distinguished three main layers in the hierarchy of M&E (sector, agency and program) and specified the program’s responsibilities with respect to each of these layers. At the program level, LJSP was responsible for M&E of the delivery of those outputs for which LJSP had accepted responsibility, and of the quality and appropriateness of its support. At the agency level LJSP was responsible for 
•	Supporting the building of agencies’ capacity to undertake planning, monitoring, evaluation and reporting as they went about their core business, whether the activities were donor funded or not; and
•	In the absence of effective agency M&E, in the short to medium term, taking a direct role in supporting agency M&E….so as to be able to report to AusAID on the impact of the Australian contribution to the sector… 
The MEF indicates that the JAG had primary responsibility for assisting the sector to measure and report on its performance, with LJSS having an increasing role.  LJSP’s role was indirect, supporting agency capacity building and information management systems. In this Framework, evaluation of capacity development emerged as a feature of program M&E in the form of a pilot capacity development study with a view to establishing whether LJSP’s support had resulted in increased capacity that was valued by Papua New Guineans. This study was still in progress at the conclusion of the program. No baseline data had been collected.