Independent Completion Report

Papua New Guinea 

Justice Advisory Group

Prepared by:
Ian Teese (consultant)
Joanne Clarkson (RPNGC)
James McGovern (consultant)

May 2009

Table of Contents
1	Background	1
1.1	Introduction	1
1.2	Policy and Program Setting	1
1.3	Project Formulation and Design	1
1.4	The JAG Program	2
1.5	Evaluation Process	3

2	Performance against JAG Key Performance Indicators	4

3	Relevance	5
3.1	Objectives	5
3.2	Activity Design and Links to the Law and Justice Sector	5

4	Effectiveness	6
4.1	Achievement of Objectives	6
4.2	Standard of Outputs	9
4.3	Benefits to Law and Justice Sector Stakeholders	10

5	Efficiency	10
5.1	Timeliness and Appropriateness	10
5.2	Value for Money	10
5.3	Implementation	12
5.4	AusAID Management and Monitoring	12
5.5	Sector Stakeholder Monitoring	13

6	Sustainability	14
6.1	Institutional Capacity	14
6.2	Recurrent costs	15

7	Crosscutting Issues	16

8	Monitoring and Evaluation	17
8.1	JAG M&E Processes	17
8.2	JAG Performance Assessment	17
8.3	Contribution to M&E Processes and Outputs	17

9	Analysis and Learning	17

10	Impact	17
10.1	Contribution to GOPNG and GOA Sector Policies	17
10.2	Law and Justice Sector Processes	18

11	Advisory Group Modality	18
11.1	JAG Implementation	18
11.2	Other Monitoring Modalities	19
11.3	Future Advisory / Monitoring Activities	20

12	Conclusions, Recommendations and Lessons Learned	21
12.1	Overall Assessment	21
12.2	Conclusions and Recommendations	22
12.3	Lessons and Good Practice	24

List of Tables

Table 1	Activities and Resourcing During Each JAG Phase:	3
Table 2	Performance Against JAG KPIs	4
Table 3	Analysis of Tasks Commissioned	9
Table 4 	Analysis of JAG Expenditure	11
Table 5	 Elements of a Continuous Learning Monitoring / Supervision Process	20
Table 6	 Evaluation Ratings at Completion	21

Appendices
Appendix 1 	Terms of Reference Independent Completion Review, Justice Advisory Group 
Appendix 2	List of Meetings and People Consulted
Appendix 3 	Tasks Completed by JAG
Appendix 4 	Draft Aide Memoire Presented
Appendix 5 	Consolidated Approved and Actual JAG Costs
Appendix 6	Feedback on JAG Implementation and Impact
Appendix 7	Monitoring and Supervision Modalities
Appendix 8	Continuous Improvement in Program and Project Implementation
Appendix 9	Lessons and Recommendations from Final Report on Monitoring and Evaluation Aspects of AusAID Assistance to Papua New Guinea’s Law and Justice Sector, 2003 – 2007.  Jessica Kenway

Abbreviations and Acronyms
AMC	Australian Managing Contractor	
AMT	Activity Management Team	
ACR		Activity Completion Report	
APR	Annual Performance Report	
ASF	Advisory Support Facility	
ATL	Australian Team Leader	
AusAID	Australian Agency for International Development	
CCS	Community Crime Survey	
CJLU	Community Justice Liaison Unit	
CIMC     	Consultative Implementation Monitoring Committee	
CPA	Contractor performance assessment (of LJSP contractor)	
CS	Department of Correctional Services	
CSO	Civil society organizations	
DB	Development Budget	
DJAG	Papua New Guinea Department of Justice and Attorney General	
DNPM	Department of National Planning and Monitoring	
DNPRD	Department of National Planning and Rural Development	
ECP	Enhanced Cooperation Program	
F/T	Full time	
GoA	Government of Australia	
GoPNG	Government of Papua New Guinea	
ICR	Independent completion review (and report)	
IMMETWG	Information Management Monitoring & Evaluation Technical Working Group	
JAG	Justice Advisory Group	
JPD	JAG project director	
KPI	Key performance indicators	
LJSP	Law and Justice Sector Program	
LJSS	Law and justice Sector Secretariat	
LJSWG	Law and Justice Sector Working Group	
M&E	Monitoring and Evaluation	
MEF	(Law and Justice Sector Program) Monitoring and Evaluation Framework ( LJSPMEF)	
MTDS	Medium Term Development Strategy	
MTR	Mid term review	
NCD	National Capitol District	
NCM	National Coordinating Mechanism	
NEC	National Executive Council	
NRI	National Research Institute	
PALJSP	PNG-Australia Law and Justice Partnership	
PDD	Program Design Document	
PFM	Peace Foundation Melanesia	
PME	Program monitoring and evaluation	
PFD	Project Formulation Document	
PMF	Performance Monitoring Framework	
QA	Quality assurance	
RPNGC	Royal Papua New Guinea Constabulary	
SHW	Stakeholder Workshop	
SMR	Six Monthly Report	
SOS	Scope of services	
SSF       	Sector Strategic Framework	
S/T	Short term	
TOR	Terms of Reference	

Basic Project Data, Acknowledgement and Disclaimer:

Country:	Papua New Guinea
Activity Name:	Papua New Guinea Justice Advisory Group
Program:	Bilateral
Location of Activity:	National level, with research and community crime surveys conducted in Provincial locations
Counterpart Agencies: 	National Judicial Services, National Judicial Staff Services, Department of Justice and Attorney General, Royal Papua New Guinea Constabulary, Correctional Services, Public Prosecutor’s Office, Public Solicitor’s Office, Magisterial Services, Ombudsman Commission and the Department of National Planning and Monitoring
Managing Contractor: 	Educo Pty Ltd
ICR Team Members:	Mr.  Ian Teese (Team Leader); Chief Inspector Joanne Clarkson (RPNGC); Mr.  James Mc Govern, Law and Justice Specialist.  Ms.  Joanna Houghton, PNG Section, AusAID Canberra, accompanied the ICR Team in PNG.  

Project Area:

(Note: This information replaces a map of PNG image that is available in the PDF version of this report)

Papua New Guinea
West Sepik province: Vanimo
East Sepik province: Wewak
Madang province: Madang
Enga province: Wabag
Western Highlands province: Mount Hagen
Southern Highlands province: Mendi
Kundiawa province
Eastern Highlands province: Goroka
Morobe province: Lae
Gulf province: Kerema
Western province: Daru
Central province: Port Moresby
Northern province: Popondetta
Milne Bay: Alotau
West New Britain province: Kimbe
East New Britain province: Rabaul
New Ireland province: Kavieng
Manus province: Lorengau

Key Dates:
5 October 2002	Request for Tender	
20 January 2003	Educo Pty Ltd awarded JAG management contractor	
20 January 2003	Phase A commences	
November 2004	Mid term review and performance review	
January 2005	Full time PME adviser and data analyst appointed	
June 2005	Following May 2005 ‘summit’, draft Roles and Responsibilities for Information Management and M&E in PNG LJS prepared	
January 2006	Full time PME officer appointed	
January 2006	Phase B commences	
February 2006	Second Performance Review and Assessment	
January 2007	PME adviser moves to part time inputs	
December 2007	Review of Sector Use of Performance Information and Sector APR (Staples) 	
January 2008	Phase C commences	
January 2008	Third Performance Review and Assessment	
January 2009	JAG completed	

Approved Cost of Activity as per Subsidiary Arrangement:
Government of Australia			AUD $ 13.787 million

Acknowledgements and Disclaimer:

The ICR Team would like to thank all those consulted for giving their time to provide valuable advice and analysis.  Special thanks are also extended to AusAID staff in at the Australian High Commission in Port Moresby and in Canberra whose support allowed the mission to proceed smoothly.

This report only reflects the views of the ICR Team.  It does not necessarily reflect the views of the Government of Australia, the Government of Papua New Guinea or any of the agencies or people consulted during the mission.

Executive Summary

1.	In January, 2009, the Government of Australia (GoA), through AusAID, and the Government of Papua New Guinea (GoPNG) completed a Justice Advisory Group (JAG) activity as part of assistance to PNG’s law and justice sector which includes the sector wide PNG Law and Justice Sector Program (LJSP).  An independent completion review (ICR) was undertaken in country, in addition to the required ICR tasks, the team assessed the JAG modality.  
2.	The Sector is one of the GoPNG’s strongest as stakeholders have developed a coherent sector approach across two branches of government (the executive and the judiciary).  
Conclusions 
3.	Impact: The JAG supported justice agencies and AusAID during the transition to a sector-wide program approach to law and justice sector development.  Sector stakeholders valued highly the JAG’s independent strategic observations, advice and information which provided sector agencies and GoA with policy options and performance data not previously available.  The community crime surveys, annual performance reports and strategic policy perspectives contributed to more effective sector planning, budgeting and implementation.  Development of these outputs and outcomes into sustained impacts is the task of the new PNG-Australia Law and Justice Partnership (PALJP).  
4.	JAG outcomes contributed to GoA country and sector strategies and aligned with donor harmonisation efforts under the Paris Declaration and PNG Declaration on Aid Effectiveness.
5.	Relevance: The JAG activity described in the tender scope of services was appropriate and relevant to the early stages of the sector program.  
6.	Effectiveness: The JAG provided valued contestible policy advice.  Some confusion about the role of the JAG, relative to the LJSP, reduced its effectiveness, particularly after the first 18 months as most stakeholders tended to view the JAG as a support facility for AusAID, rather than a sector-wide resource.  
7.	Efficiency: Considering its broadened mandate, flexibility and the nature of specific tasks undertaken, the JAG provided value for money.  
8.	Sustainability: Due to weak links to relevant GoPNG institutions, such as the Department of National Planning and Monitoring (DNPM), and its non-GoPNG institutional status, which excludes it from the GoPNG recurrent budget, the Law and Justice Sector Secretariat (LJSS) will need ongoing capacity building to fully implement the JAG functions it took over.  
9.	Despite the complexities associated with mainstreaming gender within a Melanesian context and in the absence of a specific gender-related tasking, the JAG highlighted the importance and benefits of disaggregated data for sex and age to the sector agencies’ planning processes.   
Recommendations
10.	Recommendations are numbered as in the detailed recommendations set out in Section 12.2.  Lessons and good practices are presented in Section 12.3
Independence
11.	Recommendation #2: The National Coordination (NCM) and Law and Justice Working Group (LJSWG) develop processes for contracting independent collection, collation and analysis and presentation of the crime survey data, preparation of the draft APR, at least, one more review of the Development Budgeting process.
12.	Recommendation #4: In 2010, the NCM, with the LJSWG, lead a study of sector stakeholders to assess access to and the quality of strategic policy advice, including the effectiveness of GoA support to enable GoPNG access to independent, strategic policy advice.  

National staff and consultants
13.	Recommendation #6: As recommended in the PALJP design, additional efforts should be made to identify suitable national technical specialists and develop their capacity to progressively replace internationally sourced technical assistance in the Sector.  For sector and program performance monitoring this will require accessing existing local expertise and partnerships with local research organisations.  
Lessons and Good Practice
14.	Sector coordination support: Sector-wide support mechanisms, such as the LJSS in PNG, need to be both sector and government driven to afford sustainability.  Structured formal links to the national planning and budgeting processes and coordination agencies together with an understanding of partner country institutional structures, including leadership, mandate and capacities are needed.  These would enhance sector cooperation and the sustainability of an integrated sector development approach and is more aligned with the Paris harmonisation principles.
15.	Resourcing flexibility: AusAID needs to improve its capacity to identify and then manage needed changes in program leadership and/or implementation support and resources.  This function will become more significant as AusAID becomes increasingly integrated into program and project management through Post and Desk officers, technical area advisers and technical advisers recruited directly to work in or lead sector programs.  
16.	Functions of advisory / monitoring group: A challenge for implementing advisory inputs to monitor projects and programs is how to balance the monitoring function with designing program and sector performance M&E processes and systems.  Development and initial implementation of sector and program M&E processes should be the responsibility of the program implementation contractor (as in the Transport Sector Support Program) when one program covers all development activities in a sector.
17.	However, where several donors are supporting programs and projects within a sector (such as the PNG health sector), there would be benefits in having a single facility similar to the JAG with full time resources to lead and support development and implementation of the sector M&E processes and, subject to donor funding and requirements, also support development of the individual program and/or project M&E activities.  A high level sector management or technical adviser could be based in the facility and take a leading role in sector advisory and monitoring missions.  
18.	In both situations, all stakeholders, particularly the sector coordination group, have to have direct input into the supervision and management of the advisory group including developing terms of reference for each input or specified task and ensuring agreed recommendations are followed up and implemented where appropriate.
19.	Program monitoring approach:	Monitoring support to development programs / projects should be structured as learning processes, rather than audits and/or contractor performance assessments, in which the stakeholders have significant inputs to the design, implementation and final outcomes of each review which should focus on a small number of priority issues decided by the NCM with AusAID.
20.	Policy and program coherence: Where AusAID engages an advisory group separately to a program, primary responsibility for ensuring policy and program coherence rests with AusAID, which needs to be appropriately resourced to play this role.   

1	Background
1.1	Introduction 
1.	AusAID and the Government of Papua New Guinea (GoPNG) implemented a Justice Advisory Group (JAG) activity as part of a broader program of assistance to PNG’s law and justice sector (the Sector) including the PNG Law and Justice Sector Program (LJSP).  The JAG inputs were completed at the end January 2009.  AusAID quality and development effectiveness assessment processes require that programs should be assessed at completion through the independent completion report (ICR) process.  This report documents the review undertaken by independent consultants.
1.2	Policy and Program Setting 
2.	The law and justice sector is a unique sector of the GoPNG.  Unlike other PNG Government sectors such as health, education, mining, and transport, the law and justice sector is made up of separate agencies across two branches of government (the executive and the judiciary), many with constitutional independence, but bound together by the operation of the justice system and the delivery of other multi-agency operations and strategic objectives of the law and justice sector in PNG.  Prior to the Sector approach, coordination between the law and justice agencies had been very limited and cooperation had been limited largely to interaction through cases in the criminal justice system.  Planning of the sectoral approach identified areas for cooperation and coordination, which, if improved, would lead to an overall improvement in the delivery of law and justice services to the community across PNG.
3.	Early in 2003, with support from the Justice Advisory Group (JAG), the PNG law and justice sector collaborated to establish a National Coordination Mechanism (NCM), which had been initially described within the 2001 PNG Law and Justice Sector National Policy and Plan of Action.  The NCM was intended to function as a coordinating body between sector agencies, a forum in which agencies could discuss sector issues and identify strategies to improve delivery of law and justice services.  
4.	The Sector welcomed support from the JAG, and later the LJSP, as it established its coordination processes, particularly processes relating to planning and funding activities under the development budget process for allocating funds to priority agency/sector activities.  
1.3	Project Formulation and Design 
5.	The request for the JAG project followed a joint review of the law and justice sector which led to agreement between GoPNG and GoA to identify and adopt a sectoral approach to law and justice in PNG.  At that time, law and justice sector institutional strengthening projects based in individual law and justice sector agencies were concluding and sector agencies had agreed that future development support would be provided through a sector wide integrated program.  In 2003, AusAID development support was moving from individual agency capacity development projects to a single sector delivery strategy.  This change of delivery methodology marked a major change in process for the sector and AusAID.  
6.	To support this new approach, AusAID and GoPNG agreed to establish a shared resource of independent expertise to support policy development and assist with monitoring and evaluation (M&E).  The resulting JAG activity differed from previous M&E processes, in that it provided both the GoPNG and AusAID with access to a panel of experts who could be tasked with analysing and reporting on issues identified as being critical within the sector or LJSP.  The JAG would also develop the capacity of the law and justice sector to monitor and report on sector performance as an input to improved sector strategic and implementation planning processes.  The panel was to be supported by an in-country secretariat.
7.	Although developed before the Australian Development Cooperation Strategy 2006-2010, the sector wide approach for the law and justice sector aligned with the priority core areas later specified in: (i) improved government and government building; and, (ii) improved service delivery and stability and followed three of the guiding principles.  The design also aligned with donor harmonisation efforts, which culminated in the Paris Declaration and were later more specifically described in the PNG Declaration on Aid Effectiveness.
1.4	The JAG Program 
1.4.1	Objectives and Scope 
8.	The objectives and scope of the JAG were to:
(a)	Support the GoPNG in its role of conducting the law and justice sector’ monitoring and evaluation (M&E) by providing independent advice to both GoPNG and to AusAID on the performance of the law and justice sector, including the impact and outcomes of donor and government funding and the links to poverty reduction;
(b)	Advise on any policy, structural, financial or other issues for the sector, which may include advice on broader social, political and economic conditions, policies and/or institutions outside the formal law and justice framework;
(c)	Provide specialist technical advice in relation to policy, management and/or operational matters, including in the context of specific AusAID activities in the sector;
(d)	Assist in the promotion of sector coordination through a consistent and collaborative approach to sector monitoring, in the development of agreed sector outcomes and indicators, and in the collection of sector performance information;
(e)	Support the building of GoPNG capacity to undertake sectoral monitoring; and,
(f)	Monitor and evaluate managing contractor performance in relation to the proposed AusAID-funded LJSP.
1.4.2	Implementation Arrangements 
9.	In January 2003, AusAID engaged Educo Pty Ltd (Educo) as managing contractor for the JAG.  Unlike other advisory groups, Educo provided a full time team leader and support staff based in Port Moresby who were initially supported by a part time project technical director and a panel of technical specialists who could be tasked to undertake short term inputs as requested by GoPNG and AusAID.  To strengthen the performance monitoring and assessment aspects of the JAG, full time program monitoring and evaluation (PME) consultants and officers, and a data analyst joined the PNG based team from early 2005.
10.	The activity was implemented in three phases: from 20th January 2003 to 19th January 2006 (Phase A); 20th January 2006 to 19th January 2008 (Phase B); and, 20th January 2008 to 19th January 2009 (Phase C).  The JAG operated for 6 years with a one year contract extension ending on the 19th January 2009.  
11.	The JAG implementation contract was based on payment for management services and contracted consultants to undertake agreed tasks plus milestone payments for required planning and implementation reports.  A small allocation (2.7 % of the total contract or 4.4 % of non adviser costs) was paid on a performance basis based on achievement of agreed key performance indicators (KPIs) relating to the JAG objectives and implementation performance.  These KPIs are listed in Section 2.
1.4.3	Activities and Resourcing
12.	Table 1 sets out the main activities, including AusAID and the GoPNG commissioned tasks, and the resources provided through the JAG.

Table 1	Activities and Resourcing during Each JAG Phase:
Phases	Main Activities	Resourcing	
Phase 1
Jan 2003
To Jan 2006	Providing high level strategic advice to LJS
Establishing JAG office
Creating awareness of JAG role and functions
Providing resources and secretariat to support NCM and LJSWG
Commence development of PMF and collecting LJS performance data through IMMETWF
Appraising LJSP design documents, reports and annual plans
Undertake 19 tasks (in addition to CPAs)
Undertaking LJSP contractor assessment	Technical project director – major short term (S/T) inputs
Full time (F/T) Team leader
F/T Head of secretariat, Sector liaison officer
F/T Administration officer
F/T M&E specialist (2005 only)
S/T M&E specialists (up to 2005)
S/T Task consultants	
Phase 2
Jan 2006
To
Jan 2008	Providing high level strategic advice to LJS (reduced S/T inputs)
Transferring secretariat role to LJSS
(Reducing) technical inputs to NCM and LJSWG
Complete development of PMF
Undertake 5 tasks plus 3 CPAs 
Undertake community crime surveys and other surveys on crime
Prepare, present and publish APR
Appraising LJSP reports and annual plans	Technical project director – 
F/T Team leader
F/T, Sector liaison officer
F/T Administration officer
F/T M&E specialist (2006 then ½ time)
F/T data analyst / M&E specialist 
F/T M&E officer
S/T Task consultants	
Phase 3
Jan 2008
To
Jan 2009	Transferring secretariat role to LJSS
(Reducing) technical inputs to NCM and LJSWG
Complete development of PMF
Undertake community crime surveys and other surveys on crime
Undertake 3 tasks plus 1 CPA
Prepare improved APR
Appraising LJSP reports and annual plans	F/T Team leader
F/T Head of secretariat, Sector liaison officer
F/T Administration officer
F/T data analyst/M&E specialist 
F/T M&E officer
S/T Task consultants	
1.5	Evaluation Process
1.5.1	Aims of ICR
13.	As stated in the terms of reference (TOR) provided in Appendix 1, the aims were to:
(a)	evaluate the performance of the JAG; 
(b)	assess the JAG model through rigorous ‘proof of concept’ testing;
(c)	enable AusAID and GoPNG to reflect and act on the lessons from the JAG;
(d)	inform the design and implementation of future assistance to improve AusAID’s ability to meet GoPNG development challenges; and
(e)	build evidence and learning to support AusAID’s Annual Review of Development Effectiveness report, Annual Thematic Performance Reports, Annual Program Performance Reports and Country/regional strategy reviews.
1.5.2	ICR Approach and Methodology
14.	The ICR Team used three main approaches:
(a)	A review of documents relating to JAG outputs and implementation (listed in Annex to Appendix 1);
(b)	Focused discussions with key informants (meetings and participants listed in Appendix 2); and,
(c)	A short survey of law and justice sector and JAG stakeholders to provide some structured assessment of the effectiveness and impacts of the JAG and to provide an opportunity for formal feedback.  The feedback tool and analysis are provided in Appendix 5.

15.	The draft Aide Memoire presented to the stakeholder meetings at the end of the in-country meetings is attached in Appendix 4.
1.	
2	Performance against JAG Key Performance Indicators
16.	A table summarising the JAG’s performance against key performance indicators is set out below.
Table 2	Performance against JAG KPIs
	Assessment (5 point scale)	Comments by ICR Team	
	MTR	Second 2006	Third
2008	ICR		
KPI 1: Substantive advice provided to the GoPNG, AusAID, and Sector agencies is independent, relevant and (where appropriate) includes advice on the likely impact of funding on the sector and poverty reduction.	S	A
(60%)	A
(75%)	A	The JAG provided substantive advice in sector M&E throughout the contract.  This was initially difficult due to absence of a sector strategic framework.  Stakeholders regarded highly the higher-level strategic and policy advice provided in the first half of the JAG.  The scope of the JAG’s strategic observations was narrowed during the second half of the JAG, with emphasis more on strengthening the GoPNG/AusAID understanding and capacity to undertake sectoral monitoring.  The JAG was to focus less on provision of advice on substantive law and justice policy issues within the sector due to concerns about the lack of GoPNG ownership of the generation of the advice, the pace at which the JAG produced such advice, and a feeling that the policy formulation role of the JAG was inhibiting opportunities for justice agencies themselves to identify and generate policy options.  Nonetheless, many stakeholders noted the absence of the JAG’s strategic inputs, and while the pace and scope of its inputs may have been stifling, an unrealised potential to stimulate policy debate may have reduced its overall benefits.	
KPI 2: Procedural achievement of outputs in Annual Work plan	S	FS
(100 %)	FS	FS	The presentation standard of the APR improved in the last 12 months making it more user-friendly.  An on-going weakness was that ideas for tasks identified by stakeholders were not developed by the JAG, nor sufficiently supported by the Program, lessening the JAG’s impact.	
KPI 3: JAG develops consistent and collaborative monitoring and evaluation principles and procedures that lead to agreed sector outcomes and indicators and collection of sector performance information.	S	A
(77%)	A
(77%)	G	JAG developed consistent M&E principles and procedures that lead to agreed sector outcomes; greater collaboration to realise agreed indicators and collection of sector performance information could have occurred.  	
KPI 4: GoPNG and AusAID understanding of the sector and capacity to undertake sectoral monitoring enhanced.	S	G
(80%)	G
(80%)	G	During Phase A, the JAG provided valuable contributions to GoPNG’s and AusAID’s understanding of the Sector and to Sector monitoring outcomes.  As the LJSP became more established and the MTR de-emphasised high level strategic advice, opportunities for the JAG to contribute to Sector understanding withered.  The independent focus on sectoral monitoring was a valuable contribution.  Transfer to the LJSS of conduct of community crime surveys and collation of data from stakeholders into the APR was sound, but the capacity of the LJSS to undertake these tasks needs further development; GoPNG capacity to undertake sectoral monitoring is not yet assured.	
KPI 5: JAG monitoring and evaluation reports are acceptable to GoPNG and AusAID.	S	G	G
(77%)	G	An effective process for assessing and documenting sector performance was developed (with an evolving process to improve presentation of the information).  The processes to undertake the LJSP contractor performance assessment (CPA) evolved into a satisfactory annual process..	
KPI 6: Educo standard of services based on contract requirement.  	S	FS	FS	FS	Educo delivered the required services and inputs within an evolving sector environment, where the LJSP provided implementation support 	
Overall
	S	G	G	G	During Phase A, the JAG exceeded its planned outputs and impacts.  The changes following the MTR and the consolidation of the LJSP management and advisory resources reduced the impact of Phase B and C.  	
Assessment scoring: Unsatisfactory (0-49%) - US, Satisfactory (50-69%) - S, Good (70-84%) - G, Very Good (85-99%) - VG, and Excellent (100%) - E.  (From first assessment); 	Overall very weak, 20 %, Weak 40 %, Adequate: 60 %, Good: 80 %, Fully Satisfactory: 100 % (Used for following assessments)

3	Relevance 
3.1	Objectives 
17.	At the time the JAG activity commenced, the objectives described in the Request for Tender scope of services (SOS) were highly relevant to the stage of development of the Sector.  The SOS highlighted areas where the JAG was to operate in a way substantially different to previous AusAID-funded monitoring and review groups in the Sector.  The JAG was to:
(ii)	Be a shared GoPNG/AusAID resource; 
(iii)	Provide sectoral information and policy advice; and,
(iv)	Implement program monitoring and evaluation.
18.	An implicit objective (reported to the ICR team by some key participants in the JAG design and early implementation) was to introduce ‘contestability’ in the provision of high-level policy advice to the LJSP.  Led by the JAG project director (), the JAG provided a range of high level policy and strategic advice, which was valued by the GoPNG stakeholders.
19.	During the ICR discussions, some AusAID managers commented about the degree to which the JAG activity contributed positively to sector cohesion.  It is noted that AusAID, as donor, appeared to take primary responsibility for managing two management contractors operating in the same sector.  Both stakeholders and AusAID representatives indicated that the quality of technical inputs into strategic observations and policy advice was good, particularly before the MTR.
20.	Swift mobilisation of the JAG contributed to thinking about the sector’s priorities and approaches to addressing needs.  Independent and different perspectives on sector performance and priorities were valued and fed into planning and implementation, particularly in the early years.  Stakeholders valued the advice provided by the JAG on broader social, political and economic conditions and broader GoPNG policies affecting the Sector.  GoPNG stakeholders appreciated the value to the sector’s development of this on-partisan advice and the strategic observations.
3.2	Activity Design and Links to the Law and Justice Sector
21.	The JAG process defined in the initial SOS was relevant and appropriately described.  The initial concept of a full time team leader, splitting their time between PNG and their home base, to lead and provide advice on sectoral performance assessment and high level strategic advice supported by a national managing a full-time in-country presence was an appropriate way to balance the costs of attracting advisers with the required high level skills with maintaining a permanent in-country presence.
22.	The design did not clearly indicate how the JAG and the much larger LJSP would interact in areas where there was potential for confusion and overlap.  No formal processes (regular meetings, issue resolution) were documented.  This led to an ongoing role for AusAID in mediating where overlaps and misunderstandings were creating problems.  AusAID led a process of ‘summits’ and meetings of the leaders of both activities to mediate the differences.  To the credit of the three parties, most of these differences created minimal issues to the stakeholders.

4	Effectiveness 
4.1	Achievement of Objectives 
Planned
23.	Provision of contestible policy and strategic advice: The JAG achieved in part its objective of providing independent contestible advice to GoPNG and GoA on sector performance, including impact and outcomes of donor and government funding and the links to poverty reduction.  A key strength of the JAG noted by stakeholders (and lamented in its absence) was the provision of contestible advice and strategic observations on the Sector.  Stakeholders reported that the JAG’s sector research was useful and that it has had some sustainable impact, in that it is still cited as references for the Sector.  The quality of the advice was improved by the performance information being collated during design and implementation of the Performance Monitoring Framework (PMF) and production of the annual performance reports (APR).
24.	Strategic advice also appears to have played an important role for sector managers, particularly in the early years of the JAG activity, by providing them with alternate views, some of which differed from accepted assumptions and approaches (on the part of both the GoA and the GoPNG).  Several respondents commented that at the time they found the differing ideas and presentations challenging but in hindsight, the different views helped to develop their awareness and understanding of the issues and improved their professional skills.
25.	Although the scope of the JAG policy advisory role was reduced and made more focused following the MTR, and the partial transfer of that role to the LJSP, stakeholders were very clear about the difference between the type and quality of policy/strategic observations provided by the JAG and those provided by the LJSP, generally viewing the latter as largely GoA and/or contractor driven.  
26.	Presentation of alternative high level policy and strategic advice: The issue of the style of presentation and delivery of alternative high-level interaction and advice was raised with the ICR Team.  Influence on high-level policy and strategy development is greatly improved, and probably more effective, when undertaken using a low profile, consultative approach to develop and maintain sound relationships with the key senior stakeholders (members of the sector coordination group).  
27.	Monitoring sector performance: The JAG, with the law and justice sector stakeholders, developed an effective sector Performance Monitoring Framework (PMF) and reporting process.  An early issue was that there was no sector strategic framework (SSF) around which to develop the PMF.  While steps were taken to align these two frameworks, a more coordinated approach would have reduced confusion for sector stakeholders and led to greater ownership of the resulting processes and indicators.  An ongoing issue was the differences in assessing sector, agency and LJSP performance.  
28.	Support for building of GoPNG capacity to undertake sector monitoring was less effective.  With sector agreement, the JAG assumed this function, which more appropriately might have been housed in the Department of National Planning and Monitoring (DNPM).  Capacity and political challenges in DNPM inhibited the JAG’s (and more broadly, GoA) efforts to build capacity within that institution; there was also a desire on the part of the Sector to ‘own’ its own sector M&E processes.
29.	While the JAG played a constructive role in sector monitoring; the contribution of the JAG in supporting the building of GoPNG capacity to conduct this function is less clear.  In part this is due to the sector’s decision to establish the Law and Justice Sector Secretariat (LJSS) independent of existing GoPNG structures but accountable to the NCM and linked to the DNPM (also see discussion in 6.1.1).  Significant work remains to transition the LJSS roles and responsibilities into an organisation fully funded by GoPNG.  
30.	Crime incidence data:	 While developing the PMF, the JAG identified that consistent data were not available on the incidence of crime across PNG.  The JAG developed and implemented the process of community crime surveys (CCS) to provide law and justice agencies with reliable data on perceptions of crime and the degree of crime victimisation experienced within each community, collected and analysed independently.  The data collected added to the limited data collected by agencies and increased the perspective and knowledge of the extent of crime in a particular urban area.  
31.	Sixteen urban crime surveys were completed between 2004 and 2008, supported by business crime victimisation surveys in the National Capital District (NCD) and in Lae.  The CCS followed a schedule set by the NCM.  Initially the CCS were implemented through the National Research Institute (NRI) until it withdrew from the CCS process in 2007 and the LJSS and JAG jointly managed the CCS, with the JAG assisting the LJSS to build its CCS managerial capacity, leading to a transfer of management to the LJSS in early 2008.
32.	The law and justice sector agencies have increasingly accepted the veracity and usefulness of the data over time.  The data were available to assist the sector with recurrent and development budget planning and funding requests and provided information on crime victimisation within selected urban communities.  The RPNGC used CCS data to develop its growth plan for 2008-2016, which the National Executive Council approved in 2008.  The data are available to provincial law and justice committees through the sector provincial engagement framework.  
33.	Following requests for more user-friendly presentations (also see next section), the LJSS provided participants with colour brochures highlighting the survey findings and used survey data to increase public awareness through newspaper articles and supplements, radio talkback shows and public meetings.
34.	Annual Performance Report: To provide a report on progress against the Sector PMF, the annual performance report (APR) was developed.  Due to the staged development of the PMF, as the LJSP developed the SSF based on initial work undertaken by the JAG, the contents of the APR also changed and the SSF and PMF evolved to an agreed law and justice sector format.  As a result of improved planning by agencies and the Sector, the APR is now aligned more closely with the SSF.  At the same time, sector agencies improved the quality, extent and timeliness of the data collected as they became more aware of the benefits of collecting and using better quality data for planning and reporting processes.
35.	As the APR evolved, it became larger and more difficult to read and interpret and therefore less useful to stakeholders.  Consultation with the stakeholders did not resolve the issue so AusAID tasked a Review of Sector Use of Performance Information and the APR (Staples, 2008).  Based on this report and its own experience, the JAG developed a more user-friendly format which received greater acceptance and use by the Sector.  APR data have been used by agencies to a greater degree than in the past.  The RPNGC required additional copies and the report was used at station commander workshops in Port Moresby and provincial commander workshops in Kokopo.  The National Judicial Staff Services and Magisterial Services reported greater use of the 2007 APR in the development of project formulation requests through the development budget cycle.
36.	The JAG continued to build capacity in the LJSS to produce the APR supporting the LJSP inputs.  Significant ongoing support, including a closer alignment with and input by GoPNG institutions, such as the DNPM, into the APR process appears likely.  The JAG has made a significant early contribution to this important activity.
37.	LJSP contractor performance assessment (CPA): The LJSP CPA process had a mixed start with the first two assessments conducted in the context of a change in LJSP contractor leadership and management, an evolving methodology, and criticisms that the reviewers were not independent, as they had undertaken some tasks for the JAG and had possible conflicts of interest.  
38.	These issues were largely resolved by 2006 through the JAG contracting perceived ‘independent’ evaluation specialists to perform the CPA, even though at least one member of the CPA team who was ‘acceptable’ to the LJSP contractor was also undertaking tasks for the JAG.  When the same core group of consultants was contracted each year to implement the CPA, the CPA process appeared to run more smoothly, although there were surprising negative reactions in the 2006 review, conducted by the same team as the previous and following reviews.  As noted later, the CPA process became a major activity, which may not have added value to the law and justice sector activities.  
39.	Appraisal of LJSP contractor documentation: The JAG team had an ongoing role in appraising design, annual planning and report documents and other documentation developed by the LJSP contractor and referred to the JAG.  This was a major ongoing task particularly in the early stages.  Amongst other issues, the JAG appraisals highlighted the weaknesses in the LJSP M&E framework (MEF) and areas where the MEF could be improved.
Unforeseen Outcomes
40.	Prompt response to sector priorities: An unforeseen (or at least not explicitly stated in the design) benefit was that the JAG was able to respond quickly to address emerging sector issues and coordination when the LJSP could not provide that support.  The JAG is generally viewed as having played a constructive role in supporting the establishment of conventions for sector coordination; the JAG was less successful in transitioning this support to GoPNG institutions, although political and budgetary issues also affected this outcome.  The ICR Team noted some mandate creep on the part of the JAG in performing this function.  The expansion of the JAG’s mandate beyond that envisaged in the original SOS, including assumption of the sector secretariat function, while creating efficiencies and stimulating sector progress, nonetheless captured some sector responsibilities which are being devolved back to the GoPNG with some difficulties.
41.	Secretariat to NCM: The quality and level of service provided by the JAG in its secretariat function to the NCM and the LJSWG was highly praised.  The LJSS was established independently of the DNPM, (although the Secretary of DNPM was the original chair of the NCM).  Stakeholders indicated that establishing the LJSS outside GoPNG institutions made it more effective and contributed to sector tangible advances but acknowledged that its current non-institutional status provides challenges in terms of sustainability.
42.	Location of the Law and Justice Secretariat: Prior to the JAG, AusAID through the Advisory Support Facility (ASF) had been supporting the DNPM to strengthen the group responsible for monitoring the law and justice sector.  However, the ASF inputs did not generate strengthened links between the law and justice sector and the DNPM.  The JAG’s provision of sector secretariat services on an interim basis and subsequent transfer of that function (and implementation of the sector performance framework) to the LJSS supported through the LJSP has also not been successful in improving sector linkages with the DNPM.
4.2	Standard of Outputs
43.	In general, the standard of JAG outputs was high.  High quality strategic thinking task consultants were mostly engaged.  The PMF and APRs were generally seen as of high quality, if initially, user-unfriendly.  Significant reports which were specifically mentioned during the ICR mission included:
(i)	the 2004 Restorative Justice Report, which detailed the foundations of a proposed restorative justice model, including the interconnectedness of community and state-centred justice initiatives; 
(ii)	the 2004 Village Courts Report (Goddard Report), which evaluated the impact of earlier projects and the existing system and produced a series of recommendations to improve capacity building, training and resourcing of Village Courts; and,
(iii)	“Fighting Corruption and Promoting Integrity in Public Life in PNG” (the Barnett Report) which significantly influenced changes in the law and justice sector approach to anti corruption and integrity.  
44.	A major contributing factor to the high standard of outputs was the quality assurance role undertaken by the JAG project director (JPD).  The JPDs put much effort into this role but these efforts may have diverted attention and efforts away from the just as important issue of ensuring that the task and M&E reports were acted on and extended to the law and justice sector.  These responsibilities were not explicitly outlined in the SOS nor included in later contract amendments.
45.	Development of the Sector PMF:	 The JAG was tasked to develop and implement a Sector PMF as a priority activity.  Sequencing of mobilisation of the JAG and the absence of a SSF on which to build the PMF resulted in delay in the implementation of this priority activity.  Additionally, the JAG had difficulties due to availability and timing in mobilising nominated short-term consultants with the relevant M&E capacity to develop the PMF.
46.	Utilisation of JAG resources: Sector use of the JAG’s capacity to undertake priority tasks was low, with only 6 out of 32 taskings initiated by the GoPNG agencies, as set out in the table below.  A summary listing of taskings is provided in Appendix 5.  In the last three years the JAG had difficulties generating tasks under its mandate with only nine activities tasked, (with at least four of these being AusAID specific taskings).  In Phases B and C, the JAG’s role was skewed toward supporting GoA priority tasks in the Sector.  Stakeholders and the JAG members themselves indicated that they provided suggestions for taskings, but these did not lead to completed tasks and recommendations.  
47.	
Table 3	Analysis of Tasks Commissioned (consultant estimates)
Commissioned by:			
AusAID 27		GOPNG 5 (6)	
Tasks Relating To:		Comments	
Sector performance assessment 	7	plus APR and crime incidence and impact studies	
LJSP implementation	8		
LJSP contractor performance assessment	4		
Law and justice sector studies	4	Including Barnett Corruption Study	
RPNGCDP support and assessments	6		
Other	3	ECP briefing, UNDP Safer Cities, Peace Foundation	
 
48.	Stakeholders reported that some JAG recommendations were not progressed by the LJSP.  Although some sector requests were beyond the JAG’s mandate, more could have been done by the LJSP, (with program coherence provided by AusAID), to initiate and facilitate development of JAG tasking requests from sector agencies (through the NCM), which would have generated GoPNG ownership and could then have been synergistically supported by the LJSP.   
4.3	Benefits to Law and Justice Sector Stakeholders 
49.	The JAG provided valuable independent support to the Sector during the transition to a sector-wide program approach to law and justice sector development.  Community crime surveys, annual performance reports, independent reviews of the annual Development Budgeting process, and strategic policy perspectives were highly valued outputs which contributed to more effective sector planning, budgeting and implementation outcomes.
50.	The JAG played an important role in supporting the emergence of sector-wide approaches and strategic thinking.  In a Sector which requires engagement across two branches of government, including stakeholders holding constitutional posts and, in particular, with members of the judiciary and magistracy who guard perceptions of independence fiercely, this is a significant achievement.  
51.	Nonetheless, most GoPNG stakeholders did not initially know and/or clearly understand the reasons for the JAG, its functions and its role with respect to the LJSP contractor.  While not a major constraint in the first 18 months when the LJSP was being upscaled, this contributed to the JAG being viewed as a support facility for AusAID, more than a sector-wide resource.  This partially explains its under-utilisation as a sector resource, particularly later in the JAG life.  
5	Efficiency 
5.1	Timeliness and Appropriateness
52.	Stakeholders considered the JAG’s swift mobilisation as timely, as it met emerging stakeholder needs for increased sector cohesion, through provision of support to development and consolidation of sector coordination mechanisms and collation and reporting of sector wide data.  
5.2	Value for Money
53.	The JAG provided reasonable value for money taking account of the extension of the JAG’s mandate to include provision of secretariat services to the NCM prior to the sector’s establishment of the LJSS considering flexibility afforded by the JAG and specific tasks undertaken.  
5.2.1	Contracting Process
54.	Contracting JAG prior to LJSP contractor: Contracting the smaller JAG activity before the much larger LJSP implementation contract was put out to tender may have reduced the potential number of contractors interested in tendering for the JAG activity as the successful tenderer would have been excluded from the much higher value (and total margin) LJSP contract.  
55.	This lack of competition may have not have increased the unit rates for the Australian managing contractor (AMC) inputs but may have limited submission of alternative approaches to resourcing and staffing the JAG to reduce the overheads of a full time in-country presence.
56.	Changes to Request for Tender SOS: The acceptance of the alternative proposal with inputs from a highly skilled and qualified technical project director significantly increased the potential cost of the JAG and created some duplication with the roles and responsibilities of the JAG team leader.  These changes may have been justified if the inclusion of the JAG technical project director had met (or was supported by short-term inputs) the requirements identified in the SOS.  The original SOS requested that the team leader provide the high level strategic and policy inputs in addition to the sector performance monitoring framework development skills which were to be supported by short-term M&E advisers, where necessary.  
57.	Hindsight suggests that better value may have been obtained if the original SOS had been followed and AusAID had recruited any additional high level policy and strategy advice resources required through the JAG, ASF or period offer processes.  This would have provided the flexibility to change these inputs in response to changing technical or project related situations and decoupled the high level advice from the management of a support facility.
5.2.2	Implementation of Justice Advisory Group
58.	The following table provides an analysis of expenditure over the life of the JAG.  The data are based on the total contracted value and the actual (IT estimates for final Phase C) expenditure during the three contract phases of the JAG activities.  These phases also align with the evolving changes in the roles and responsibilities of the JAG in response to the mid term review (MTR) and development of the LJSP which were outlined in Table 2.
Table 4 Analysis of JAG Expenditure		
Item Description	Value as per contract	Phase A	Phase B	Phase C	Total	% of	
		2003-2005	2006-2007	2008**	Expenditure	Expenditure	
Management Fees	 	 	 	 	 	 	
Milestone Payments	2,694,107	1,199,669	979,061	515,378	2,694,108	23%	
Performance Payments	299,345	127,672	89,747	50,822	268,241	2%	
Subtotal	2,993,452	1,327,341	1,068,808	565,198	2,962,347	26%	
JAG Technical Inputs	 	 	 	 	 	 	
Technical Project Director	1,091,060	617,738	287,907	2,533	908,178	8%	
JAG advisers: secretariat based**	 	250,000	812,500	250,000	1,312,500	11%	
Task** JAG Advisers and sub-contracts incl.  travel	6,553,408	2,194,015	1,094,542	838,808	4,127,365	35%	
Subtotal	7,644,468	3,061,753	2,194,949	1,091,341	6,348,043	54%	
Team Leader	1,476,264	712,097	468,904	267,994	1,448,995	12%	
Secretariat Costs: admin	1,156,153	299,037	190,256	94,115	583,408	5%	
Sector engagement **	 	150,000	132,000	72,000	354,000	3%	
Other Expenses 
(NCM costs, publications, etc)	517,335	7,156	7,645	6,205	21,006	0%	
Secretariat and LJSS Support Costs	3,149,752	1,168,290	798,805	440,314	2,407,409	21%	
TOTAL CONTRACT 	13,787,672	5,557,384	4,062,562	2,097,853	11,717,799	100%	
Note: ** detailed breakdowns are estimates by ICR team	
59.	The table shows that more than 60 % of expenditure (increasing to 70% if management fees are pro-rated across expenditure areas) was on tasks that were needed as part of the development of the LJSP.  Stakeholders reported that development of the PMF and associated processes were enhanced by the appointment of the long-term M&E specialists and project officers who provided a consistent approach and methodology to a process that created some tensions within the Sector.  
60.	The main areas where value for money may have been reduced include:
(a)	Duplication: The duplication in technical responsibilities between the technical project director and team leader roles.  Also the resourcing of both the LJSP and the JAG to provide strategic policy advice – a role which might have been more cost effectively limited to one of the two contractors.
(b)	Reduced JAG activities: The reduced level of JAG activities in Phases B and C when the secretariat responsibilities of the JAG were being transferred to the LJSS and the role of providing strategic policy services was taken over by the LJSP.  Estimated annual JAG secretariat costs (including team leader costs) did not reduce significantly in Phase C even though the law and justice sector performance monitoring and reporting functions were being transferred to the LJSS.
(c)	Payment of management fees: The basis of payment of the management fees was mainly based on milestone payments.  As taskings for the budgeted professional inputs were not implemented and the management fees remained fixed, the AMC was effectively receiving a margin for all potential inputs.  The ICR team is confident that the AMC made their best efforts to encourage additional taskings.
61.	Resourcing flexibility: A further area where Value for Money could have been improved would have been in modifying the inputs of advisers, sector engagement officer, the team leader and technical project director as circumstances and sector needs changed.  Stakeholder feedback indicates that later in the project, the inputs and roles of senior JAG management could have been changed to improve efficiency, but AusAID contract management processes could (did) not lead to major changes.
62.	This reluctance / inability to facilitate changes to resourcing highlights a major issue that will become more significant as AusAID becomes more integrated into program and project management through Post and Desk officers, technical area advisers and technical advisers recruited directly to work in or lead sector programs 
5.2.3	Relative to Other Advisory Modalities
63.	The discussion in 5.2.2 highlights that the major factor influencing Value for Money in the JAG is the level of management overheads to support the PMF, tasking and CPA processes that would have been undertaken regardless of the delivery modality (JAG, period offers, ASF and similar).  
5.3	Implementation
64.	The efficiency of the JAG (and LJSP) start-up was reduced by:
•	An inadequate communication program at the start of the two activities so stakeholders understood the different roles for the JAG and the LJSP;
•	personality conflicts;
•	changes in team leadership; and,
•	a staggered start of the LJSP creating a perception that LJSP responsibilities had been ceded to the JAG.  
65.	This combination resulted in differences between the JAG and LJSP staff on roles and responsibilities, and substantial management inputs by AusAID Post staff, which created inefficiencies.  Without the need for AusAID to devote effort and time to resolving territorial disputes, it could have more usefully focussed its efforts on supporting the GoPNG to realise its strategic direction for the Sector.  
66.	Issues raised by the JAG in ‘front-end’ activities (implementing tasks, producing performance data) received insufficient ‘rear-end’ LJSP support.  Efforts were made to put in place an agreed framework for operations and approach, but the disconnect continued to the end of the JAG.  The JAG’s attempts to ensure that the LJSP could follow up on the review recommendations were not fully successful and required additional inputs by the Post.  
67.	A more constructive relationship between the JAG and the LJSP staff would have streamlined support to the Sector (and made management and supervision by AusAID and the Sector much easier).  The two groups of staff could have worked more closely together without affecting the perceived independence of the JAG in its contractor assessment role.  This required a proactive management positioning to realise program and policy coherence in the Sector.
5.4	AusAID Management and Monitoring
68.	Interviews with Port Moresby-based AusAID staff responsible for oversight of the JAG activity, and with JAG Team Leaders, revealed a perception that AusAID had expended a disproportionate amount of time and effort on JAG supervision and management, partially to clarify the emerging roles and responsibilities between JAG and the LJSP contractors, and also to address areas where one or both parties raised concerns.  Despite this, the JAG was clearly seen as a resource that senior AusAID representatives could bring to the table in discussions with GoPNG about the law and justice sector.  
69.	AusAID used ‘summits’ to facilitate mediation / allocation processes between the JAG and LJSP staff, however AusAID also continued to have ongoing significant management inputs even after the issues of overlapping responsibilities between contractors had apparently been resolved.  Some of these ongoing inputs appear to have been personality driven, while others appear to have focussed on ongoing discussions about the role of the JAG, vis à vis the LJSP.  Some conflicts may have been based on contractual issues.  As the much larger LJSP scaled up, the focus of AusAID’s management inputs shifted to the LJSP, and the tension between the JAG and the LJSP contractors continued to create issues for the AusAID program managers, although these were reduced following the MTR.
70.	The levels of AusAID input into both the JAG and the LJSP required to align GoA's policy and program approach to the Sector were underestimated.  The need to maximise perceptions of independence for the JAG may also have played a role.  There was scope for further attention by the LJSP and by AusAID to engender ownership and leadership by GoPNG in the JAG policy products and recommendations, under a suitably resourced coherent program and policy approach.
71.	An intended JAG function was to provide a level of contestability to LJSP approaches and sector assumptions about law and justice priorities and issues.  Harnessing that contestability to benefit the development of the sector requires significant coherence of approach between GoPNG and GoA over the contractors for the JAG and for the LJSP.  
5.5	Sector Stakeholder Monitoring
72.	The GoPNG sector stakeholders had only indirect inputs to monitoring of JAG activities.  None of the stakeholders interviewed raised issues of JAG under-performance.  The main issues raised by stakeholders included:
•	The focus of the JAG on delivering a functioning PMF and annual reporting processes (which were JAG payment milestones) which led to pressure on the agencies to provide performance information without a clear explanation of why the information was needed and how it would be used.  This pressure may have contributed to a major issue at one NCM meeting when the RPNGC representatives disputed data presented in the APR.  Feedback from stakeholders indicates the issue related more to the JAG processes in collecting and using the data than the actual data; and,
•	The issue identified earlier of a lack of awareness of the purposes and role and mandate of the JAG.  This could have been addressed through a more targeted communication strategy, including awareness and sensitization, supported by AusAID.  Further, the JAG could have focussed its work on culturally sensitive ownership building strategies, which might also have assisted stakeholders’ understanding of its role and benefits for the law and justice sector.
73.	The ICR Team observes that there were no formal joint GoPNG-GoA JAG management and coordination mechanisms, such as a project steering or coordination committee to allow GoPNG inputs to the supervision and monitoring of the JAG.  These inputs could be made informally through the NCM but the ICR Team believes that, without a formal monitoring role, the NCM and LJSWG were pre-occupied with the start-up of the much larger LJSP following on from the discontinued agency focused projects.  Despite the intended ‘light footprint’ of the JAG, a more structured JAG management and coordination mechanism, involving all stakeholders may have increased the number of priority tasks implemented in the last 2-3 years and assisted AusAID manage the professional tensions between the JAG and LJSP contractor management teams.
74.	 An issue raised by non-GoPNG stakeholders was that having two sources of high level law and justice sector policy and strategic advice (contestability) created confusion for the GoPNG stakeholders.  The GoPNG stakeholders interviewed did not complain of confusion from differing advice and observed that in situations when differing views were put at a sector meeting, the stakeholders were quite capable of firstly, asking the proponents to remain objective, and then making their own judgements on the appropriate outcomes.  

6	Sustainability 
75.	The sustainability of the sector-wide performance reporting processes may be limited under the LJSS which is still building its management and technical capacity and is currently not well integrated within GoPNG bureaucratic structures.  A closer, structured relationship with DNPM could have institutional advantages.  Ongoing capacity building, as proposed in the PALJP design, in appropriate GoPNG institutions is required to embed the independent sector support provided by the JAG.
6.1	Institutional Capacity 
6.1.1	Secretariat Functions
76.	The transfer of secretariat responsibilities to the LJSS from 2006 proceeded smoothly but the CSS and APR preparation and reporting processes are still evolving and require further development of LJSS capacity, raising questions of sustainability.  Some reservations were noted in the Sector (and shared by the ICR Team) about the capacity and sustainability of the LJSS, which was broadly perceived to be a part of (and is substantially funded through) the LJSP.  The LJSS exists outside the GoPNG institutions, although it has adopted GoPNG administrative and financial systems, presumably to align with GoPNG processes.   Initial efforts to base the LJSS within the DNPM were not successful due in part to capacity shortfalls and to an absence of consensus about whether the secretariat should be aligned to the sector or a central agency.  
77.	The LJSS was established independent of DNPM, (although the Secretary for DNPM was the original chair of the NCM and, as such, endorsed the approach that was adopted).  While stakeholders indicated that establishing the LJSS as an entity autonomous from DNPM made it more effective, and allowed the Sector to make some tangible advances, they acknowledge that its current non-institutional status remains a challenge for the Sector in terms of sustainability.  The LJSS was established for a trial period subject to review.
78.	The sustainability of the secretariat function performed by the JAG and now transferred to the LJSS will depend in part on the strength of evolving formal and informal links to the DNPM (and to other GoPNG agencies with a mandated role in the law and justice sector planning and budgeting processes).  Further dialogue between the DNPM and the Sector and its agencies is needed on the future roles and activities of the LJSS, including whether the LJSS is aligned with a GoPNG agency within or outside of the Sector, and whether the LJSS has access to funding through GoPNG’s recurrent budget processes.  Ongoing GoA budgetary support will be required to sustain this function.  The prospects for sustainability are also connected to the professionalism and levels and quality of service provided by the LJSS to sector stakeholders, which stakeholders noted have reduced since the handover from the JAG.
6.1.2	Annual Performance Reports
79.	Given its current capacity levels, the ability and appropriateness of the LJSS proactively extending its scope beyond secretariat to the NCM and LJSWG, for example to conduct or manage the conduct of research and sector performance appraisal, such as the APR remains unclear.  While the JAG worked to transfer this function to the LJSS, ongoing support will be required to develop the LJSS’ capacity to perform the duties expected of it.  Further discussion of what those duties are is also required.  
80.	The ICR Team notes that the new PNG-Australia Law and Justice Partnership (PALJP), specifically acknowledges the importance of independent advice and contemplates ongoing support for:
“Planning for, accessing and managing external and independent evaluation and advisory services.  This might include support for partnerships with local research institutions and the use of private consultancies – in both cases drawing and building on established relationships of this kind within the sector.  This will be critical to enable the sector to further develop and refine the functioning of its Performance Monitoring Framework and related assessment tools.”	
81.	The ICR Team is encouraged by the PALJP’s continuing support for the Sector to produce its APR.  Significant ongoing support, including a closer alignment with and input by GoPNG institutions, such as the DNPM, into the APR process appears likely.  The JAG has made a significant early contribution to this important activity.  
6.1.3	Sector Research and Strategic Observations
82.	As noted above, the PALJP specifically includes provision for ongoing independent advice and collation of sector performance data.  This is to be supported by working closely with the LJSS and DNPM to determine and plan for particular requirements annually.  
83.	The ICR Team is also encouraged by the support anticipated for additional services required to support the LJSS and/or DNPM in undertaking community crime surveys, and for the Sector to undertake targeted, longitudinal impact studies in selected SSF result areas.  The centrality of research and sector performance appraisal and the focus on planning and implementation of programs and activities, such as sector relevant research and appraisal under recurrent as well as development budgets anticipated under the new PALJP, will complement the JAG’s legacy in the Sector.
6.1.4	Monitoring Development Budget Preparation Process
84.	Stakeholders highlighted the benefits of the JAG in monitoring and providing independent feedback on the Development Budget process.  Most stakeholders believed that the independent monitoring should continue for at least another year.  Ongoing support is included in the upcoming PALJP for support for independent reviews of the sector’s annual development budget process.  Currently, responsibility for continuing the process monitoring function has not been allocated.  The role has been undertaken by a JAG M&E specialist who is available for the next review so there is probably little advantage in training a LJSS staff member or local consultant to undertake the task for one more budget cycle in the law and justice sector.  Development of DNPM Capacity
85.	The JAG was not able to work closely with the DNPM to strengthen monitoring capacity within DNPM, which has primary responsibility for monitoring achievement of objectives of the MTDS.  While part of the solution to strengthening program monitoring capacity within DNPM are internal administrative and resource changes, the JAG and the GoA may have been able to provide assistance through programs outside the Sector such as democratic governance programs to build key law and justice sector monitoring capacity within DNPM.  This could include working through other projects and non-GoA funded development partners working in the field in PNG.
6.2	Recurrent costs 
86.	Recurrent costs associated with the various successor functions of JAG activity continue to fall largely to the GoA, and are spread between the LJSS’ budget (funding received by LJSS through the Development Budget, which, in 2009, is predominantly GoA funded) and the PALJP.  A challenge to sustainability of the LJSS (and the whole law and justice sector partnership) will be securing access for LJSS funding through GoPNG’s recurrent budget processes.  The ICR team understands that the sector proposals for GoPNG contributions to the 2009 development budget were greatly reduced.  The scope of JAG-like activities that need to be incorporated into ongoing law and justice sector programs (and where) and the scale of GoPNG recurrent funding attributed to those activities needs to be determined.

7	Crosscutting Issues 
87.	The nature of the JAG and its functions meant that gender and HIV awareness were the main crosscutting issues it could address.  Stakeholders interviewed indicated that despite the difficulties of progressing gender equality in a Melanesian cultural context, the JAG contributed to raising gender awareness, through its requests from sector agencies for sex and age disaggregated data.  The ICR Team notes that the JAG and its contractor do not appear to have adopted a gender mainstreaming approach to the JAG’s activities, and that neither government tasked the JAG to undertake priority gender related tasks, such as studies on the issue of sorcery and sorcery related murders, which have a high proportion of women victims.  The ICR Team notes, however, that aspects of gender were integrated into some of the JAG’s taskings.  Areas where the JAG took account of, or addressed gender included:
•	Community Crime Surveys provided gender disaggregated data on crime.  This data provided useful insights.  For example, it revealed that women tended to be victims of men known to them (especially relatives and neighbours rather than spouses) and that domestic violence and sexual assault were special cases affecting women; 
•	The issue of sorcery and sorcery related murders and the high incidence of women victims.  In PNG this is a gender issue, as most accused sorcerers are women and violence against these women is commonplace.  The JAG raised the issue of gender and sorcery with the Chief Magistrate and other NCM members on an ongoing basis in order to encourage NCM and law and justice sector action to address this predominantly gender based crime;
•	The 2005 APR identified the need to recruit more female village court magistrates (previously recommended in the first Village Court review) and highlighted the sector’s inability to provide data disaggregated on the basis of gender.  This led to a proactive plan to increase the number of female village court magistrates from about 60 in 2004 to more than 270 in 2008; 
•	The 2006 APR highlighted that no progress had been made in developing information systems within the law and justice sector to allow performance data to be collected distinctly about men and women, different age groups, or different locations.  The APR also discussed poverty reduction and the need to be able to link activities within the law and justice sector to poverty reduction, and highlighted weaknesses in some sector agencies in the development of HIV/AIDS plans.
•	The improved 2007 APR format highlighted the impact of crime on women and the improvement in law and justice sector and agency responses to family and sexual violence.  In addition the APR highlighted those law and justice sector agencies that provided human rights services in terms of women and children’s rights under the Constitution.  The APR also provided details of agencies that have HIV/AIDS workplace policies in place which are clearly linked to agency core business; and
•	The ICR Team notes that the JAG core team of the JAG was predominately male.  However, the JAG actively developed the capacity of female PNG development practitioners as a contribution towards establishing a cohort of professional women consultants who could act as role models for other PNG women.  
88.	There appears to have been unrealised potential by the JAG to progress gender equality in PNG.  Discussions with the Gender and HIV advisers in the LJSP suggest that this issue of integrating the crosscutting issues into law and justice sector programs is an ongoing task, which will require further specialist inputs and targeted support from GoA through the PALJP.
8	Monitoring and Evaluation
8.1	JAG M&E Processes
89.	The JAG implemented the reporting processes documented in the SOS and following contract and amendments.  A regular schedule of meetings with AusAID was followed up with structured meetings to assess JAG performance.  The close interaction between the JAG and AusAID in designing the PMF and the agreed tasks provided good informal interaction in the first phase.  This may have been reduced in the following phases as communications were then focused through the team leader.  
8.2	JAG Performance Assessment
90.	The JAG developed a set of KPIs (see Table 1) for assessing its own performance.  These were reported on annually and used in the three independent assessments of the JAG performance.  The independent assessments elaborated on the KPI framework and quantified the assessment against the performance indicators to contribute to the final assessment of contractor performance by AusAID and agreement on the performance fee to be paid.  
8.3	Contribution to M&E Processes and Outputs
91.	The JAG filled an unavoidable gap in development of the law and justice sector M&E processes as GoPNG’s sector approach was evolving and developing before the LJSP contractor was appointed.  The tasking of the JAG to support GoPNG to develop a PMF was appropriate but became complicated due to the lack of a SSF to develop the PMF around.  
92.	Until the LJSP contractor modified its approach to LJSP M&E in the second half of the LJSP, the JAG guided development of the SSF to align with the PMF.  The JAG provided a focus on sector wide M&E issues.
9	Analysis and Learning
See Conclusions, Recommendations and Lessons.
10	Impact 
10.1	Contribution to GOPNG and GOA Sector Policies 
93.	The JAG activity was broadly perceived by sector stakeholders as making a positive contribution to development of the law and justice sector.  In particular, the JAG’s collation of information, and provision of advice and options to both governments were valued for their perceived independence, quality and for the fact that they provided alternative views and perspectives on issues.  It was noted that this on occasion challenged sector agencies’ accepted views, including about the type of data collected through internal M&E systems, and that working through the alternative views presented by the JAG was not always easy.  Sector stakeholders reported that the Sector would not have realised its current achievements in the absence of those challenges presented to the Sector and to individual agencies by the JAG’s alternate perspectives and data, which ultimately were seen as assisting stakeholders in their decision-making processes regarding Sector and agency priorities.
94.	In line with the above, the JAG’s generation of sector performance information and APR were, and appear to continue to be, valuable to the Sector.  The JAG was faced with an initial lack of understanding of the need for the basic agency data and worked to address capacity challenges within each contributing agency regarding collection and collation of data.  The APR produced by the JAG provided core information about the Sector, and over time the APR was presented in a more user-friendly way.  Community crime surveys and community crime data collected through CSOs were valuable additions to the resources available to the Sector for policy formulation, planning, and budget implementation through both the development and recurrent budget processes.
95.	Over time, stakeholders increasingly accepted the data generated through JAG processes as identifying areas that required the attention of sector agencies, including in policy and implementation.  Respect for the data was built through the (independent) JAG, together with the LJSP team and GoA, shepherding a process of challenge, discussion and review of the data with sector agencies.  While this process was at times challenging for individual sector agencies, data outcomes generated changes in policy as noted in Section 4.1.
96.	The JAG’s independent monitoring of and reporting back to the Sector on the annual Development Budget preparation process was seen as increasing efficiency and effectiveness of that process, with the JAG able to provide a sounding board to individual agencies and to the Sector as a whole to discuss priorities.  Again, the fact that this function was provided by an independent third party was highly valued.
10.2	Law and Justice Sector Processes 
97.	Annual Performance Report: The development and implementation of community crime surveys and the systematic reporting of these results with other appropriate agency resource, output and outcome data provided the Sector with its own credible independent source of sector performance data.  This information is being used by sector agencies to redirect resources within each agency.  Feedback to the ICR Team indicated that the Sector wanted the APR process continued and the presentation of the information improved to make it more usable by senior decision makers and politicians.
98.	Secretariat to the NCM: Stakeholders valued the early support of the JAG as secretariat to the Sector, to the NCM, and to the LJSWG.  Sector feedback was that the JAG resources balanced excellent secretariat support services with provision of an alternative source of ideas and advice to sector agencies.  
99.	The transfer of secretariat responsibilities to the LJSS proceeded smoothly but some of the other functions previously implemented by the JAG appear not to have had a smooth transition.  Some reservations were noted in the Sector (and shared by the ICR team) about the capacity and sustainability of the LJSS.  Sustainability is likely to be improved by several factors including:
(a)	Evolving formal and informal links to the DNPM;
(b)	The ability and appropriateness of the LJSS proactively extending its functions beyond acting as secretariat to the NCM and LJSWG, for example to conduct or manage the conduct of research and sector performance appraisal;
(c)	Securing recurrent GoPNG funding; and
(d)	The PALJP’s adoption of a whole-of-budget approach to supporting the sector, in particular for planning and implementation of programs and activities under both recurrent as well as development budgets, may result in increased relevance accorded to the LJSS, and maximise its sustainability.  
11	Advisory Group Modality
11.1	JAG Implementation
100.	Contracting the JAG separately to the LJSP provided GoPNG and GoA with distinct advantages as both governments had access to contestible well respected information and views about the Sector, its performance, and for specific issues, such as development budget process monitoring.  
101.	An initial strength of the JAG modality was its permanent presence on the ground in PNG, which GoPNG stakeholders largely saw as having great merit in that it permitted continuous engagement by the Sector with the JAG, including the background and understanding of issues and of participants in the Sector, and opportunities to develop relationships.  Stakeholders valued this level of knowledge and access over the fly-in fly out modality of an independent monitoring group, such as that used in the PNG health sector, which was reported to have members with varied levels of knowledge and experience about the PNG health sector and poor quality stakeholder relationships, and was proving unwieldy for both GoPNG and GoA managers.  
102.	Separate contracting of the JAG to the LJSP, while appropriate from the perspective of assuring maximum independence for the monitoring and advisory functions, and mobilisation of the JAG contractor before the LJSP contractor set up a management dynamic which proved difficult for GoA to manage.  The TOR for the JAG was vague on the extent to which the JAG was required to contribute positively to sector policy and program coherence.  
103.	In the JAG situation where major tasks were to develop program and sector monitoring processes, separating the development of the processes from the program contractor monitoring processes was achieved by using different consultants managed by the same team leader.  This was a practical outcome in the situation where the main LJSP contractor selection and mobilisation process was lagging behind mobilising of the JAG team.  However, there are no intrinsic benefits (and has shown in the LJSP, some negatives) to having the sector and program M&E frameworks and processes developed separately from the program as ensuring smooth interaction between the two groups complicates design and implementation.  However, in the case of the LJSP, given the issues in the LJSP team, the LJSP contractor probably could not have completed the work any faster than the JAG team.
11.2	Other Monitoring Modalities
104.	The experience summarized in Appendix 7 is relevant to PNG as overall government structures and processes are similar in most countries with ongoing issues on the role (and power) of the central government budget and activity coordination and planning agencies (DNPM and the Central Agencies Coordination Committee in PNG).  The situation in PNG is easier to work with as the central agencies (despite their acknowledged weaknesses) have the leading role.  In other countries where GoA is a development donor such as the People’s Republic of China, Viet Nam and Indonesia (to some degree), implementation of sector and program M&E activities is becoming much more difficult because of the devolution of power and responsibilities to the provincial level.
Papua New Guinea
105.	A comparison of project and program monitoring modalities used in Papua New Guinea including experience from other developing countries is provided in Appendix 7.  Many monitoring activities are seen as an audit of project inputs, activities and outputs and can apply to program contractor performance assessment.  For projects funded by development lending agencies (and some bilateral donors), a more comprehensive process usually called supervision missions are used as they usually also monitor the use of loan funds.  
106.	Health sector: The monitoring model currently used in GoPNG and GOA/other donor funded health sector program is an advisory group(funded by sector donors) designed to focus on higher level issues but becomes involved in operational issues aligning with team member interests.  Discussions with Australian stakeholders suggest the monitoring group is not working effectively for reasons including: (i) weak linkages to the sector coordination group; (ii) a changing group of high level specialists with a range of relevant PNG experience; and, (iii) no clear process to prioritise and then implement recommendations.  The emphasis is on technical issues rather than on implementation contractor assessment.  For GoA funded projects, this contractor assessment is undertaken by external consultants who, on at least one health sector program, implemented this as an operational audit focusing on one area during each six monthly input led by the same team leader.  This was found to be a very constructive process.
107.	Transport Sector: GoPNG and AusAID are still developing the monitoring processes for the Transport Sector Support Program.  The Program contractor is developing the sector and program M&E processes but no decision has been made yet on which GoPNG agency should take long term responsibility for these functions to support the sector coordination group.  High level strategic advice is provided to the sector coordination group by the project director contracted directly by AusAID.  The Program also has a strategic adviser working in the relevant section of the DNPM.  
Cambodia Justice Sector Project
108.	The Cambodian Criminal Justice Advisory Group (CCJAG) provides six monthly short term inputs for external review and monitoring of the GoA supported Cambodian Criminal Justice Advisory Project (CCJAP).  The CCJAG uses parts of models 1 and 2 set out in Appendix 7 providing short-term strategic inputs, using national and international consultants with detailed knowledge of the Cambodian Justice Sector.  The CCJAG also contributes to contractor performance assessments, The CCJAG has flexibility to evaluate other initiatives within the Cambodian Criminal Justice Sector, looking beyond individual project interventions to how GoA’s combined assistance is contributing to improvements to Cambodia’s criminal justice system as a whole.  
109.	Mission terms of reference are developed in collaboration with stakeholders on a mission by mission basis.   The third mission in October 2008 assessed progress across all CCJAP components, with a particular focus on support being delivered in provinces and capital works.  The CCJAG has also considered strategic and cross cutting issues and identified opportunities for improvement.  Feedback and reports are provided to the Project’s National Management Board, and a large percentage of the CCJAG’s recommendations are adopted by the National Management Board and to date successfully implemented by the Project.  The CCJAG does not have responsibility for development or implementation of sector or project M&E processes but provides advice to the CCJAP contractor on these issues.
11.3	Future Advisory / Monitoring Activities
110.	The objective of a program/project monitoring mechanism should be to implement a non-confrontational continuous improvement process that uses adult education processes and experience to harness the experience of national staff and their supporting advisers to develop areas and processes where program performance towards agreed objectives can be achieved.  Appendix 8 provides an outline of how a monitoring / supervision process could contribute to continuous improvement and learning through the program / project implementation cycle.  It is based on the summary of different monitoring modalities in Appendix 7.  The main elements are set out in Table 5.
Table 5		Elements of a Continuous Learning Monitoring / Supervision Process
	Element	Comments	
(i)	Shared ownership (and responsibility) for process	Sector / program coordination group oversee planning (including TORs) and are main reporting focus of monitoring group	
(iii)	Activity seen as learning process rather than auditing program implementation	Appropriate sector agency staff invited to actively participate as part of monitoring team.	
(iv)	Led as a facilitatory process by team leader with organization development / program design / M&E focus, not necessarily high-level technical skills	Team leader develops relationship with sector coordination group and key decision makers.  	
(v)	One team member should have high-level policy and strategic technical knowledge and understanding in sector.	Team leader may also have these skills.  AusAID funded sector adviser could provide these high level skills when sector contractor also provides high level strategic advice inputs through a technical director.	
(ix)	Each monitoring input focuses on small number of agreed priority areas (set by program/project coordinating group) rather than a program wide review.	Focused priority areas makes field work more manageable and allows greater inputs to priority areas.	
(x)	Monitoring Team circulate an draft aide memoire with main conclusions and priority recommendations to all stakeholders at least 36 hours before wrap-up meeting	Stakeholders accept that some draft conclusions and recommendations may need higher level agreement.	
(xii)	After agreement on aide memoire, stakeholders undertake to implement areas under their mandate	Commitment for action	

12	
Conclusions, Recommendations and Lessons Learned 
111.	The following conclusions, recommendations and lessons are based on the information collated by the ICR Team and also draw on the JAG completion report and studies on LJSP M&E activities commissioned by AusAID.  
112.	The ICR Team sought formal feedback from stakeholders in a questionnaire.  Feedback from the small number of respondents is provided in Appendix 6.  The feedback largely concurs with the findings of the ICR Team except in the assessment of the impact of the JAG on sector capacity to undertake performance assessment.  There was strong feedback that the full-time presence of the JAG was valuable.  Most respondents thought there should be greater use of national consultants and resources (see later lessons learned).
12.1	Overall Assessment 
12.1.1	Project Impact:
113.	The JAG provided valuable independent support to the Sector during the transition to a sector-wide program approach to law and justice sector development.  Community crime surveys, annual performance reports, independent reviews of the annual Development Budgeting process, and strategic policy perspectives were highly valued outcomes which contributed to more efficient sector planning, budgeting and implementation.  Agencies reported how these activities contributed to improved planning, implementation and outcomes.  The JAG focused sector agencies on the need for better quality M&E data.  
12.1.2	Project Modality:
114.	The full time presence of the JAG was justified at its inception by its nominated major functions, the proposed inputs and the limited AusAID resources to manage the transition from the sector agency based projects to the wider sector based program approach.  The flexible JAG structure gave AusAID flexibility to address issues arising during the transition and to mobilize resources to assist AusAID manage the start up of the LJSP.  
115.	After the LJSP contractor had completed the LJSP design process and established the LJSP management processes, on one view, the justification for the full-time JAG presence was reduced as the PMF design and implementation activities could have been implemented under the LJSP.   Some of the benefits of the full time presence of the JAG were also offset by the higher overhead management costs of the full time international team leaders who did not have a major technical input.   Nevertheless, permanent presence facilitated development of ongoing relationships with key sector stakeholders which were instrumental to the JAG’s outputs and outcomes, and would not have developed to the same degree under a fly-in fly-out model.   
12.1.3	Evaluation Ratings:
Table 6		Evaluation Ratings at Completion
Evaluation Criteria	Rating (1-6)	
Relevance	6	
Effectiveness	5	
Efficiency	4	
Sustainability	3	
Gender Equality	2	
Monitoring & Evaluation	5	
Analysis and Learning	5	
Rating scale: 6 = very high quality; 1 = very low quality.  
Below 4 = less than satisfactory.	
12.2	Conclusions and Recommendations 
12.2.1	Conclusions
Evaluation criteria
116.	Impact: As indicated above, the JAG provided valuable independent support to the Sector during the transition to a sector-wide program approach to law and justice sector development.  Development of the outputs and outcomes from the JAG activities into sustained impacts is the task of the new PALJP.  
117.	Relevance: The JAG process defined in the initial scope of services was relevant and appropriately described.  As the law and justice sector and LJSP have evolved, the functions initially provided through the JAG have been largely transferred to the LJSS or the LJSP.
118.	Effectiveness:
(a)	The JAG was able to respond quickly to a need to support sector coordination when the LJSP contractor was not in a position to provide that support; 
(b)	Stakeholders were not confused by the availability of contestible JAG policy advice, which was de-emphasised at the MTR.  Post MTR, stakeholders noted the reduced access to high level policy options and debate and raised concerns about future sources of this policy advice; 
(c)	Implementation of the priority activity to develop and implement a sector PMF was delayed as: (i) there was no SSF around which to build the PMF; and, (ii) the capacity to develop the PMF specified in the SOS was not included in the initial JAG team;
(d)	Most GoPNG stakeholders did not initially know and/or understand the reasons for the JAG, its functions and the links with the LJSP contractor.  While not a major constraint in the first 18 months, this contributed to the JAG being viewed as a support facility for AusAID, rather than a sector-wide resource.  This partially explains its under-utilisation as a sector resource later in the JAG life;
(e)	The JAG had difficulties in the last 2-3 years generating demand for tasks under its mandate.  Stakeholders provided suggestions but these did not lead to completed tasks and recommendations.  Some requests were beyond the JAG’s mandate but more could have been done by both the JAG and by the LJSP to initiate and facilitate development of tasking requests from the agencies; and,
(f)	The JAG advisers had more credibility in their role developing the PMF and supporting the sector secretariat (which involved long-term advisory positions) than short-term fly-in, fly-out teams as used in other technical advisory groups and their ongoing support and advice was valued as a result.  
119.	Efficiency: 
(a)	The JAG provided reasonable value for money considering the widening of its mandate, flexibility afforded by the JAG and specific tasks undertaken.  More than 70 % of funds directly supported technical inputs and completion of commissioned tasks and the JAG provided secretariat services to the NCM and LJSWG.  Efficiency would have been improved if the team leaders had provided more technical leadership in the policy advice and/or performance framework development.   Secretariat functions performed by the JAG were managed by experienced national resources;
(b)	The addition of a project director to the SOS, while providing additional highly valued technical expertise, complicated management of the JAG and was expensive; 
(c)	If the team leader role had been managed as a part time input (supported by national administrative support) with more focus on providing technical inputs and oversight, the JAG would have been more cost-effective; and,
(d)	A more constructive relationship between the JAG and the LJSP contractors and staff would have streamlined support to the Sector (and made management and supervision by AusAID and the Sector much easier).  The two contractors could have worked more closely together without affecting the perceived independence of the JAG in its contractor assessment role.
120.	Sustainability: Ongoing capacity building in appropriate GoPNG institutions is required to embed the independent sector support provided by the JAG.  The sustainability of the sector-wide performance reporting processes may be limited under the LJSS which is still building its management and technical capacity and is currently not housed under a GoPNG institution.  A closer, structured relationship with DNPM or an agreed higher level agency with planning and coordination responsibilities may have institutional advantages.  Further targeted capacity building of the LJSS remains a priority.  
121.	Gender: Despite the complexities associated with mainstreaming gender within a Melanesian context and in the absence of a specific gender-related tasking, the JAG highlighted the importance and benefits to the Sector in having access to disaggregated data for sex and age in analytical reports on the village courts and restorative justice.  Individual sector agency disaggregation of M&E data collection remains variable.  
Specific
122.	Independence:	Sector stakeholders valued highly the strategic observations, advice and information provided by the JAG, which was perceived to be independent and provided sector agencies and the GoA with a range of policy options not otherwise available.  The priority areas where information collation and analysis should be at least perceived to be independent were in the crime surveys, information collated into the APR, the review and feedback on the annual development budget process, and strategic observations and policy advice for sector development.  
123.	JAG Modality: The JAG model was very appropriate in the early development stage of the Sector and of the LJSP and is applicable to future strategy development by permitting contestability of concepts and approaches, although this can require significant AusAID management inputs.  
124.	National staff resources:	 The JAG used some national staff resources led by the national sector engagement officer, monitoring project officers and some national consultants.  Arrangements to use national research institutes for research projects were discontinued due to staff changes.  The ICR team received positive feedback on the work of these staff, some of whom now work with other projects.  
125.	Sequencing performance assessment framework development: While some stakeholders felt that the JAG’s development of the PMF prior to development of the SSF may have improved the SSF developed, having the SSF in place first would have greatly assisted development and implementation of the PMF.
126.	Tasking and using study recommendations: While the JAG was successful in making strategic policy observations and drawing priority issues to the attention of stakeholders, such as through the community crime surveys, pathways for the Sector to operationalise recommendations from JAG contracted studies were not developed nor supported by the LJSP.  Greater GoPNG understanding and ownership of JAG processes in the early stages of implementation would have improved effectiveness.
12.2.2	Recommendations
127.	These recommendations, and the Lessons and Good Practice set out below, are complemented by monitoring and evaluation specific lessons and recommendations from the report on Law and Justice Sector Monitoring and Evaluation (2003-2007) (Jessica Kenway, 2008), provided in Appendix 9.
Gender
128.	Recommendation #1: Continued targeted Program support to the NCM and the LJSWG will be required to support efforts aimed at ensuring sector agencies plan for and develop systems which generate data disaggregated for sex and age.
Independence
129.	Recommendation #2: The NCM and LJSWG develop processes for contracting independent collection, collation and analysis and presentation of the crime survey data, preparation of the draft APR, at least, one more review of the Development Budgeting process.
130.	Recommendation #3: The LJSS to supervise and manage these contracts to undertake surveys and prepare the APR but not undertake the work.  
131.	Recommendation #4:	In 2010, the NCM, with the LJSWG, lead a study of sector stakeholders to assess access to and the quality of strategic policy advice, including the effectiveness of GoA support to enable GoPNG access to independent, strategic policy advice, whether through the PNG-Australia Law and Justice Adviser or otherwise.
Future support to monitoring and evaluation
132.	Recommendation #5:	Care should be taken to ensure that stakeholders have maximum opportunity to input into TOR of the range of M&E mechanisms contemplated under the new PALJP, including the scope of additional services provided to the LJSS and to the DNPM for Community Crime Surveys; independent reviews of annual development budget process (this could also include broadened terms of reference for a whole-of-budget review); production of the APR; and any targeted longitudinal impact studies, particularly through engagement of local institutions.  It is important to ensure that AusAID’s management positioning and contractual arrangements with the managing contractor ensure that contractual incentives do not diminish opportunities for the PALJP to support capacity building, even if this results in a slowing of the pace of implementation.  
National staff and consultants
133.	Recommendation #6: As recommended in the PALJP design, additional efforts should be to identify suitable national technical specialists and develop their capacity of to progressively replace internationally sourced technical assistance in the Sector.  For sector and program performance monitoring this will require accessing existing local expertise and partnerships with local research organisations to allow them (through ongoing contracts and staff development support) to retain and utilise the specialised skills needed.  This will greatly enhance the cost-effectiveness and sustainability of the supported activities and processes.  
12.3	Lessons and Good Practice
134.	Sector coordination support: Some government sectors do not have an agreed (and funded) sector coordination and monitoring agency to: (i) provide secretariat services to the sector coordination group; (ii) develop the sector performance framework; (iii) undertake assessments of aid delivery mechanisms; and (iv) more importantly, develop and implement the independent data collection/collation systems needed to support the performance framework.  These functions need to be supported until they can be wholly transferred to the agreed sector planning and coordination agency.  Where it is not possible to identify an agreed sector planning and coordination agency, an external facility could be established to facilitate coordination, on an interim basis.  This latter option is the least preferred, and if adopted, the facility should have a national administrative team.  
135.	Sector-wide support mechanisms, such as the LJSS in PNG, need to be both sector and government driven to afford sustainability.  The LJSS is currently seen as part of the aid delivery mechanism and receives limited GoPNG budget support.  While a halfway-house approach may have been the best available option at the time in the PNG context, this approach should be avoided, as it trades long-term institutional sustainability and stakeholder ownership for an increased pace of project-driven (short-term) outputs.  Structured formal links to the national planning and budgeting processes and coordination agencies together with an understanding of partner country institutional structures, including leadership, mandate and capacities are needed.  These would enhance sector cooperation and the sustainability of an integrated sector development approach and is more aligned with the Paris harmonisation principles.
136.	Resourcing flexibility: AusAID needs to improve its capacity to identify and then manage needed changes in program leadership and/or implementation support and resources.  This function will become more significant as AusAID becomes increasingly integrated into program and project management through Post and Desk officers, technical area advisers and technical advisers recruited directly to work in or lead sector programs..  AusAID needs to improve its capacity to identify and then manage needed changes in program leadership and/or implementation support and resources.
137.	Contracting (and transparently taking the advice of) independent supervision / monitoring individuals or teams to identify where changes are needed (in addition to providing valued technical and management guidance support) would provide a means of objectively highlighting where resourcing / capacity weaknesses (excesses?) need to be addressed.  
138.	Functions of advisory / monitoring group: A challenge for implementing advisory inputs to monitor projects and programs is how to balance the monitoring function with designing program and sector performance M&E processes and systems.  In practice, development of sector and implementing M&E processes should be the responsibility of the program implementation contractor (as in the Transport Sector Support Program).  It is only in situations, such as the early stages of the LJSP where program development and implementation was evolving and there were ongoing agency specific projects, that a separate project with full-time staff and management could be justified to develop and lead implementation of the sector and program M&E processes.  This particularly applies when one program covers all sector development activities.
139.	However, where several donors are supporting programs and projects within a sector (such as the PNG health sector and, possibly, developing in the PNG transport sector), there would be benefits in having a single facility similar to the JAG with full time resources to lead and support development and implementation of the sector M&E processes and, subject to donor funding and requirements, also support development of the individual program and/or project M&E activities.  A high level sector management or technical adviser could be based in the facility and take a leading role in sector advisory and monitoring missions.  
140.	In both situations, all stakeholders, particularly the sector coordination group, need direct inputs into the supervision and management of the advisory group including developing terms of reference for each input or specified task and ensuring agreed recommendations are followed up and implemented where appropriate.
141.	Program monitoring approach:	Monitoring support to development programs / projects should be structured as learning processes, rather than audits and/or contractor performance assessments, in which the stakeholders have significant inputs to the design, implementation and final outcomes of each review which should focus on a small number of priority issues decided by the NCM with AusAID.  See Table 5 and Appendix 8 for more details.
142.	High level policy and strategy advice: In the Melanesian context, advice on policy and strategy development appears to be best adopted when offered through a low profile consultative approach, within the context of sound relationships with the key senior stakeholders (members of the sector coordination group).  The advisory role is best seen as a facilitator, with a good technical understanding, rather than as a provider of direct technical advice.  Direct technical inputs (when required) appear best provided through agreed external technical experts (the original JAG concept).  The ICR Team notes that direct employment of high level technical specialists by AusAID to lead sector programs does not align with the concept of ‘independent’ or contestible advice.
143.	Independent and contestable advice: AusAID’s direct contracting of high-level technical specialists to lead sector programs may not align with the concept of providing independence and contestability of advice, which underpinned the original JAG concept.  Contractors supporting sector programs should provide (and manage) inputs by high level strategic advisers (as technical project directors or similar roles) as requested and monitored by sector coordination groups.  
144.	Policy and program coherence: Where AusAID engages an advisory group separately to a program, primary responsibility for ensuring policy and program coherence rests with AusAID, which needs to be appropriately resourced to play this role.  Managing policy and program coherence should be seen as part of AusAID’s core business; insufficient resourcing, particularly in-country, reduces AusAID’s ability to remain proactively engaged with the contractors (or resources) to ensure their activities are complementary.  
145.	 Increased coordinated support for uptake of independent sector study recommendations and independent advice: Areas identified through ‘front end’ research and policy formulation initiated through activities such as the JAG and/or AusAID need to receive sufficient ‘back end’ programmatic support and extension to prioritise recommendations and to support further development and /or implementation of priority recommendations.  This requires a degree of coordination and program coherence, and an approach that fosters greater partner government ownership of “front end” products, that was not present in the JAG activity.
146.	Concurrent tendering of oversight activity and Program proper supported: Where a monitoring / advisory group is tendered separately to the program implementation contract, maximum interest and participation by potential bidders could be generated through sequential tendering of the program and oversight roles.  In the absence of this approach, GoA and the partner government need to closely manage contractor relationships, clearly delineate mandates, avoid mandate creep, and ensure that any later transfer of functions between entities are well facilitated.