PNG Law and Justice Sector Program

AidWorks Initiative Number INF361

INDEPENDENT COMPLETION REPORT

Authors’ Names and Organisations:
Dr Penelope Murphy, Consulting Plus Pty Ltd
Dr Heike Gramckow, The World Bank

 8 February 2010

Aid Activity Summary

Aid Activity Name	Law and Justice Sector Program	
AidWorks initiative number	INF361	
Commencement date	19 February 2002	Completion date	13 May 2009	
Total Australian $	$150,000,000.00	
Delivery organisation(s)	Government Partners were supported by a managing contractor, Cardno ACIL	
Implementing Partners	Main implementing partners were the Department of National Planning and Monitoring and core Government of PNG law and justice agencies; National Judicial Staff Services, Department of Justice and Attorney General (including the Office of the Public Prosecutor and the Office of the Public Solicitor), Royal Papua New Guinea Constabulary, Correctional Services, Magisterial Services, and the Ombudsman Commission.

The LJSP also supported partnerships between law and justice sector agencies, Civil Society Organisations and sub-national levels of government.	
Country/Region	Papua New Guinea	
Primary Sector	Law and Justice	

Acknowledgments
The Team wishes to acknowledge with appreciation the contributions of Mr Gabriel Kubul, Senior Program Officer, and Mr Jason Kalimba, Program Support Officer, Law & Justice Unit, AusAID Post for their work in setting up the program of appointments for the team and for facilitating their visit.  The Team also acknowledges all those consulted for their time and input to this review, including that of Ms Joanna Houghton, Manager, PNG Branch/ Economic & Public Sector Governance, AusAID, Canberra who accompanied the team throughout the in-country mission. Particular thanks go to Mr Clivson Philip, Registrar of District Courts, Magisterial Services, for his substantial and extremely valuable contributions to the team as the representative of the PNG Law and Justice Sector.

Author’s Details
Dr Penelope Murphy (Team Leader and Monitoring and Evaluation Specialist) is an independent consultant of Consulting Plus Pty Ltd, P.O. Box 145, Keiraville, NSW 2500, Australia.
Dr Heike Gramckow (Law and Justice Specialist) is Senior Counsel with the Justice Reform Practice Group Legal Department of the World Bank, 1818 H Street, NW, Washington, DC 20433, USA.

Table of Contents
Page No.
Acronyms and abbreviations
Definitions
Executive summary
1.	Introduction
1.1	Activity background
1.2	Evaluation objectives and questions
1.3	Evaluation scope and methods
1.4	Evaluation team
2.	Evaluation Findings
2.1	LJSP’s sectoral approach in global context
2.2	Relevance
2.3	Effectiveness
2.4	Efficiency
2.5	Impact and sustainability
2.6	Gender equality
2.7	Monitoring and evaluation
2.8	Analysis and learning
3.	Evaluation Criteria Ratings
4.	Lessons Learned and Recommendations
5.	Conclusion

Acronyms and Abbreviations

ACIL		ACIL Australia Pty Ltd
ADB		Asian Development Bank
AJDF		Access to Justice Development Fund
AJP		Access to Justice Program
AMT		Activity Management Team
APP		Annual Program Plan
AusAID		Australian Agency for International Development
CBO		Community Based Organisation
CIMC		Consultative Implementation and Monitoring Council
CJLU		Community Justice Liaison Unit (formerly Community Law and Justice Unit, CLJU)
CPGs		Commonwealth Procurement Guidelines
CS		Correctional Services
CST		Core Support Team
CV		Curriculum Vitae
DB		Development Budget
DFID		Department for International Development (UK)
DGP		Democratic Governance Program
DJAG		Department of Justice and Attorney General
DNPM		Department of National Planning and Monitoring
FSV		Family and Sexual Violence
GAD		Gender and Development 
GoA		Government of Australia
GoPNG		Government of Papua New Guinea
GoU		Government of Uganda
HR		Human Resources
ISP		Implementation Service Provider
JAG		Justice Advisory Group
JLOS		Justice Law and Order Sector
JR SWAp		Sector-wide justice reform efforts
KPI 		Key performance indicator
L&J		Law and Justice
LJSWG		Law and Justice Sector Working Group
LJS		Law and Justice Sector
LJSP		Law and Justice Sector Program
LJSS		Law and Justice Sector Secretariat
LTA		Long Term Adviser
M&E		Monitoring and Evaluation
MDG		Millennium Development Goals
MEF		Monitoring and Evaluation Framework
MS		Magisterial Service
MTDS		Medium Term Development Strategy, 2003 - 2007
NCDC		National Capital District Commission
NCM		National Coordinating Mechanism
NEC		National Executive Council
NGO		Non Government Organisation
OC		Ombudsman Commission
PALJP		Papua New Guinea-Australia Law and Justice Program
PCG		Program Coordinating Group
PDD		Program Design Document
PFD		Project Formulation Document
PMF		Performance Monitoring Framework
PNG		Papua New Guinea
PP		Public Prosecutor
PS 		Public Solicitor
RPNGC		Royal Papua New Guinea Constabulary
RPNGCDP III		Royal Papua New Guinea Constabulary Development Project (Phase III)
SG		Solicitor General
SSF		Sector Strategic Framework
STA		Short Term Adviser
SWAp		Sector-wide Approach
TA		Technical Adviser
TASP		Technical Advisory Support Panel
TOR		Terms of Reference
UNICEF		United Nations Children’s Fund
USAID		United States Agency for International Development

Definitions
This Independent Completion Report (ICR) relates to the Law and Justice Sector Program (hereafter “LJSP” or “the program”). The program is so closely inter-related with the Law and Justice Sector (hereafter “the sector”) as well as with the actions of the contractor responsible for managing the program (hereafter “the contractor”) that a distinction is often difficult. The success of the program must ultimately be assessed in terms of the program’s contribution to the outcomes of the sector, which in turn depends on the contractor’s contribution.  A discussion of the program without reference to the sector, individual agencies and/or the contractor would in many instances have little meaning and it is often the interaction among them that leads to results. In an evaluation of the program, there is a risk of conflating the terms sector, program and contractor or of slipping between them.  For the purpose of this report, the following definitions are aimed at clarifying terms and addressing the risk of conflating them.
Definition of the Sector
The term “sector” refers to the collectivity of (as distinct from the individual) law and justice agencies which together are represented by sector institutions, i.e. the National Coordination Mechanism (NCM) at the policy and strategic level, by the Law and Justice Sector Working Group (LJSWG) and the Law and Justice Sector Secretariat (LJSS) at the senior administrative and operational level and through the Activity Management Teams (AMTs) of the sector. 
In theory the sector also includes civil society and other organisations, as defined in the Program Design Document. In practice the involvement of these organisations in the sector has so far been minimal. They have largely been excluded by agencies in the law and justice sector (LJS) in planning and decision making about the development budget and excluded from the Pre-Screening Committee for the Project Formulation Documents. 
The Justice Advisory Group (JAG) played important roles in relation to the sector, but was not part of the sector or the program. An Independent Completion Report specific to the JAG was completed earlier in 2009.  The JAG is therefore not the subject of this ICR, although reference will be made to it.
Definition of the program
The “program” sits outside the sector. It comprises the sum total of resources (human, material and financial) provided by Australia to support agreed outputs.  The sector selects a wider set of outputs defined at agency or sector level that are consistent with the outcomes of the Sector Strategic Framework (SSF). The program supports an agreed sub-set of these.
Definition of the Contractor
The “contractor” is defined as the corporate entity contracted by AusAID (in this case Cardno ACIL) to design the program in conjunction with the sector and deliver resources, and includes all of its personnel that are not funded out of the program as Technical Advisers.  This definition therefore includes the contractor’s program management team and support staff deployed either within or outside of Papua New Guinea (PNG).   

Executive Summary
Background and context 
Within the global context the Law and Justice Sector Program (LJSP) marked the start of a very different and innovative approach to justice sector support. It required fundamentally different program processes, structures, operational procedures, management structures, engagement processes and monitoring and evaluation (M&E) assessment approaches from those of the past. This approach both pre-dated and aligned well with the Paris Declaration and PNG Commitment on Aid Effectiveness, and the Accra Agenda for Action. To achieve this, the LJSP design provided a flexible program approach through which assistance was delivered under PNG leadership and, increasingly, used GoPNG systems and processes. 
Summary of LJSP objectives, components and key results 
The explicit purpose of LJSP throughout its six-year term 2003 – 2009 was to develop the capacity of law and justice agencies and their partners to implement the Government of Papua New Guinea (GoPNG) National Law and Justice Policy, sector priorities and plans.  As PNG’s policy framework evolved over time, the LJSP objectives were adapted and refined to align with emerging priorities, particularly those articulated in the five goals of the 2005 Law and Justice Sector Strategic Framework (SSF): Improved Policing, Safety and Crime Prevention; Improved Access to Justice and Just Results; Improved Reconciliation, Reintegration and Deterrence; Improved Accountability and Reduced Corruption; and Improved Ability to Provide Law and Justice Services.  
Brief outline of the evaluation findings 
The evaluation findings are briefly summarised in the table below.
Evaluation Criteria Ratings
Evaluation Criteria	Rating (1-6)	Explanation	
Relevance	5	LJSP was highly relevant to higher level objectives of the aid program and to GoA and GoPNG priorities, but somewhat less relevant to the context and needs of end-users of law and justice system.	
Effectiveness	4	Overall, although effectiveness across the five program goals was mixed, the program was very successful in facilitating the development coordination structures and translating the meaning of a sector wide approach to agency planning, operations, management and policy.  	
Efficiency	3	While the efficiency of the program is assessed as satisfactory in later years, it cannot be rated as fully adequate given the deficiencies in the earlier years of the program.	
Sustainability	4	Capacities to sustain core corporate functions are expected to be sustainable in most agencies; sustainability in relation to broader program goals is questionable. All major sector operations, especially the secretariat and the CJLU, continue to be fully funded by the program and would not sustainable if funding ceased.	
Gender Equality	4	Within the PNG context, LJSP is assessed as having laid the groundwork and made a significant contribution to gender mainstreaming and advancing gender equality in PNG, except in promoting equality of decision-making between women and men.  	
Monitoring & Evaluation	2	LJSP gave little priority to its M&E system and overall the result was much less than satisfactory, except in the final 12 – 18 months of the program.	
Analysis & Learning	3	Although the design was based on previous learning and analysis, integration into the activity of learning from implementation and previous reviews was slow and tended to lack a systematic approach.	
Rating scale: 6 = very high quality; 1 = very low quality. Below 4 is less than satisfactory.
Brief outline of lessons learned 
The main lessons are summarised below, and are presented in more detail with other lessons and relevant recommendations in Section 4. Recommendations are consolidated in Annex 1. 
Service delivery focus: More visible impacts can be obtained by providing support for strengthening service delivery capacities at the same time as support for corporate capacities. This also produces more relevant engagement at each agency and across the sector. 
Common understanding of roles and capacities: Applying a sector wide approach to justice system reform requires that all involved (each agency, the implementing contractor and the funding agency) develop a shared and clear understanding of what this means for their respective roles and responsibilities. This requires taking stock regularly of the level of engagement needed and available capacities.  
Use of government systems: The use of the GoPNG Development Budget as the path through which all program activities are identified and selected for funding has proven an effective mechanism for operationalising the use of government systems in line with the Paris Declaration and the Accra Agenda for Action. This approach to using government systems tends, however, to encourage agencies to view change as applying only to their administrative rather than their core service delivery activities.
Communicating the meaning of a sector wide approach is key to ensuring that government planning and budgeting reflects and considers the needs of the sector before the needs of individual agencies.
Constraints to broad-based engagement:  The strong restorative justice emphasis of the National Law and Justice Policy implies there should be a greater focus on inclusion of civil justice issues that are so important to people throughout PNG. This will become even more important as the sector expands its efforts to reach the provinces and districts. 
Monitoring and Evaluation for service improvement: The program would have benefited from a strong early focus on strengthening agencies’ capacities to identify their target beneficiaries, evaluate their service delivery and develop and implement systems for monitoring and reporting meaningfully on their service delivery.  
Gender equality and HIV/AIDS mainstreaming: A thorough gender analysis in the design process, the recruitment of full time local personnel as gender and HIV/AIDS advisers, the identification of male champions in senior positions, and the allocation of adequate resources, were key to the mainstreaming of gender equality and HIV/AIDS in LJSP.
Analysis and Learning based on Action Learning Principles: Advisers, the agencies and the sector would have benefited from contractor leadership in modelling good analytic and learning practice based on action learning principles.  
The sustainability of the LJS Secretariat in its current form is questionable given its relatively large bureaucratic structure and lack of linkage to any existing government or sector agency.  

1.	Introduction
1.1	Activity Background
Law and justice is one of seven development priorities identified in the Medium Term Development Strategy (MTDS) of the Government of PNG (GoPNG). To support the law and justice sector the GoPNG and the Government of Australia (GoA) agreed on a sectoral approach to the provision of development assistance. This marked the start of a new and highly innovative approach to aid. Over the period 2003 to 2009, the Law and Justice Sector Program (LJSP), costing AUD 150 million, was the primary program (out of three) through which Australian financial support to the law and justice sector in Papua New Guinea (PNG) was channelled.  
The explicit purpose of LJSP throughout its six-year term was to develop the capacity of law and justice agencies and their partners to implement the GoPNG National Law and Justice Policy, sector priorities and plans.  To achieve this, the LJSP design provided a flexible program approach through which assistance would be delivered under PNG leadership and, increasingly, use GoPNG systems and processes. As PNG’s policy framework evolved over time, LJSP’s objectives were adapted and refined to align with emerging priorities, particularly those articulated in the 2005 Law and Justice Sector Strategic Framework (SSF).
In 2003, Cardno ACIL was engaged by AusAID as the Managing Contractor for the LJSP.  The activity was implemented in two Phases from 7 April 2003 to 31 December 2003 (Phase I – Design and Implementation); and from 1 January 2004 to 6 April 2008 (Phase II - Implementation), with Phase II being extended from 7 April 2008 to 6 April 2009.  The LJSP was in operation for six years and ended in April 2009. Its successor program, the Papua New Guinea – Australia Law and Justice Program (PALJP) was in the process of implementation at the time of this Report, with the same contractor as Implementation Service Provider (ISP).
1.2	Evaluation Objectives and Questions
As part of the sectoral approach, both governments agreed to measure the performance of the sector as a whole and to assess the relative impact of government and donor funding. Accordingly, AusAID commissioned this Independent Completion Report (ICR) to:
(a)	Assess the LJSP sectoral approach and evaluate the performance of the LJSP, with reference to the five DAC criteria for evaluating development assistance: relevance, efficiency, effectiveness, impact and sustainability, and the three additional AusAID criteria of monitoring and evaluation, gender equality, and analysis and learning;
(b)	Enable AusAID and GoPNG to reflect and act on the lessons from the LJSP;
(c)	Inform the design and implementation of future assistance to improve AusAID’s ability to meet GoPNG development challenges; and
(d)	Build evidence and learning to support AusAID’s Annual Review of Development Effectiveness report, Annual Thematic Performance Reports, Annual Program Performance Reports and Country/regional strategy reviews.
The objectives of the evaluation were to:
•	Assess the LJSP’s sectoral approach to the delivery of development assistance to the law and justice sector in PNG.  In particular, the technical, institutional and country specific factors influencing the performance of the LJSP should be examined together with the extent to which technical advisers have been used within the LJSP.  Broad comparisons with other law and justice approaches implemented by multilateral and Governmental development agencies should be undertaken to assess the relative effectiveness of the LJSP and to obtain a sharper profile of the strengths and weaknesses of the LJSP’s approach. 
•	Evaluate the performance of the activity against the criteria of relevance, effectiveness, efficiency, impact, sustainability, monitoring and evaluation, gender equality, HIV/AIDS, and analysis and learning, against which all AusAID activities are required to be measured at completion through the ICR process (consistent with AusAID’s current template for Independent Completion Reports).  The issues of particular significance in the LJSP activity were:
(a)	The effectiveness of the LJSP in meeting its objectives against the Outcomes and Objectives listed in Attachment D to the TOR; 
(b)	The impact of the LJSP on sector performance and the lessons that can be learned from the LJSP in this regard; 
(c)	The efficiency of the implementation of the LJSP, including the performance of the contractor in managing the LJSP activity in meeting contract service delivery objectives as listed in Attachment A to the TOR;
(d)	The sustainability of any improvements in sector performance induced by the LJSP and the factors critical to the continuity of any positive trends in sector performance; and
(e)	The degree to which gender equality and HIV/AIDS mainstreaming had been integrated into the LJSP. 
1.3	Evaluation Scope and Methods
The main methods used in this evaluation were: a review of documents relating to LJSP outputs and evaluations (see Annex 3); focused discussions and interviews with key government and non-government stakeholders, program staff and AusAID staff in Port Moresby; comparison with international experience in implementing law and justice sectoral programs; and triangulation of data from these various sources. An in-country mission was conducted from 19 July – 7 August 2009 to interview a broad range of stakeholders. Interviewees were initially selected by AusAID. The itinerary and persons consulted are detailed in Annex 4.
Two important assumptions were made that resulted in significant limitations.  The first was that National Coordinating Mechanism (NCM) and Law and Justice Sector Working Group (LJSWG) members, as key informants, would be available at times convenient to them to meet with the ICR Team during their three weeks in country.  AusAID staff at the High Commission spent a great deal of time calling, emailing, faxing and sending letters to members of the Department of National Planning and Monitoring (DNPM), NCM, LJSWG and sector for the purpose of setting up meetings with these key stakeholders.  Ultimately a quite limited response was achieved.  Agencies where no meetings were achieved were DNPM, Correctional Services (CS), Office of the Public Prosecutor (PP), and National Justice Staff Services (NJSS). However the presentation of the Aide Memoire at the conclusion of the mission was attended by four staff of DNPM, the PP and representatives of NJSS.  Those who attended were invited to forward comments to the Team.  Written comments were received from: Joanne Choe of AusAID, and Anthony Nolan of UNICEF.
The second assumption concerned provincial engagement. The Team were advised by AusAID that provincial visits were unnecessary as another team had recently visited provinces to conduct an evaluation of the three pilots of provincial engagement.  It was understood that the report of that team would be available for use by the ICR Team during their mission.  A rough draft Provincial Engagement Research Evaluation Report was received part way through the mission.  This was clearly a work in progress and lacked any overall summary, analysis and conclusions.  It was therefore of relatively little use to the ICR team. With the benefit of hindsight, it would have been valuable to include a visit to at least one of the three pilot provinces.
1.4	Evaluation Team
The evaluation team comprised two independent members: Dr Penelope Murphy, Team Leader and Monitoring & Evaluation Specialist and Dr Heike Gramckow, Law and Justice Specialist; a representative of the PNG Law and Justice Sector and member of the LJSWG Mr Clivson Philip (from 30.7.09), a representative of DNPM Ms Rose Koyama (afternoon of 31.7.09 only), and an AusAID Resource Person Ms Joanna Houghton.  The team composition provided an excellent balance between technical expertise, experience with AusAID reviews and evaluations, and local knowledge. The only limitation was that the two PNG members joined the team late in the mission.  
2.	Evaluation Findings
This section commences with an assessment of the LJSP’s sectoral approach in comparison with other law and justice sector approaches.  This is followed by seven subsections which address the first of the four aims of this ICR. These evaluate the performance of the LJSP with reference to the five DAC criteria for evaluating development assistance: relevance, efficiency, effectiveness, impact and sustainability, and the three additional AusAID criteria of: monitoring and evaluation (M&E), gender equality, and analysis and learning. The second and third aims of the ICR are addressed in Section 4: Lessons Learned and Recommendations. The compliance of the program with the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action is discussed under sections 2.2 and 2.7, and more fully in Annex 5.
2.1	LJSP’s sectoral approach in global context
Sector wide approaches to justice sector reform (as well as in many other sectors) are still experimental and cutting edge today and this was especially true at the start of this program in 2003. As in other countries, assistance to justice sector reform in PNG had previously been project based, i.e. largely focused on individual agencies. More importantly, and in line with the Paris Declaration, for the LJSP a “sector approach” was defined by AusAID as one that would both encompass most justice sector agencies and be driven and owned by the sector.  This recognised the fact that the overarching goal of achieving a sector wide approach to justice sector reform in PNG had been formulated by the sector itself even before program design activities started. Within the global context this is a very different and innovative approach and one which required fundamentally different program processes, structures, operational procedures, management structures, engagement processes and M&E assessment approaches from those of the past.
Not surprisingly neither the sector, nor the contractor nor AusAID could clearly predict or fully understand what a sector approach would mean concretely in terms of responsibilities, authorities and human resource requirements and how this would impact their operations and relations. The willingness of AusAID to take this leap of faith into exploring a very different way of engagement is remarkable, as is the sector’s early recognition of the value of taking a sector wide approach. As a result, this program has become a show case for sector wide and locally owned justice sector reform worldwide.  Considering the novelty and lack of models to compare to and learn from, it is not surprising that some contractor management and support activities and several program activities did not meet high effectiveness marks. This particular innovation challenge is recognised and weighed positively in this report.  
Judging the efficiency of a large and complex program that is operating in a challenging environment such as the LJSP is difficult.  In this case, the difficulty also lies in the fact that the program is quite experimental and that there are few relevant other experiences with which to compare. Taking a sector wide approach to justice sector reform is widely recognised as essential for successful reforms today, but this recognition was still new and evolving when the program started.  Even today research and theory to support justice sector reform in general is scant and still evolving, and more so within a development context. Moreover, examples of justice sector wide approaches that have been developing across the globe and have been documented fairly well come from countries that are quite different from PNG, from places such as Uganda and Pakistan. The very different country context and country capacities at program entry would really mean comparing ‘apples’ and ‘oranges.’  Still, a few international examples are provided to give a sense of how those programs evolved at what rate and price, what direction they took and what results they brought.  
Today we find sector wide justice reform efforts (JR SWAps) in a number of countries but with very different country capacities and conditions and with very different JR SWAp characteristics.  This includes countries like Afghanistan, an in-conflict country with significant needs and capacity shortcomings. There justice sector reform is largely supported via a multi-donor trust fund, even though not all donors participate in the main donor trust fund.  Earlier examples that started in 2001 can be found in Nigeria, where assistance was focused on selected states only, largely funded by the Department for International Development (DFID) with some assistance also provided by other donors, especially the United States Agency for International Development (USAID).  This effort was very much driven by the implementing contractors and DFID with country counterpart consultations and engagement. There was limited counterpart control and very limited donor coordination.
More appropriate comparators are probably the JR SWAps in Uganda and Pakistan – again countries that are vastly different in many respects, especially in their population size. The population of Uganda is 32.4 million, and Pakistan has roughly 175 million people.  Both programs started before the LJSP. The Justice Law and Order Sector (JLOS) in Uganda, which started in 2001, is a reform process ongoing across the entire justice sector through a SWAp to address the systemic weaknesses in the justice system. These weaknesses were compounded by lack of a clear policy framework for all the justice agencies to deliver quality justice efficiently and effectively. Roughly 27 billion Uganda Shilling (c. AUD 16 million) were spent during the first three years. This was about half of what had been made available. There was just no country capacity to absorb more support.  This program still continues and current plans extend until 2011.  Today, all funding for the agencies comprising the JLOS support a single policy and expenditure reform program under the leadership of the Government of Uganda (GoU).  Funding support is, however, more fragmented and provided by multiple donors that largely focus on individual agencies, such as DFID (police), Danida (Judiciary), the Netherlands (Directorate Public Prosecutions), Germany (Prison), and World Bank loans. It is impossible therefore to identify what the funding support level has been so far.  As such, this is also more of a project approach – bringing with it different coordination challenges. A mid-term evaluation report from 2004 indicates that similar experiences and challenges were faced as those experienced by the LJSP. There were significant difficulties with: developing and adhering to an appropriate M&E plan; getting the message out to other government agencies; and with establishing a management structure and handling different funding streams.  Importantly, however, within the first 3 years of operations the program achieved some measurable impact on service delivery as a result of a strong focus on assistance for improved frontline operations.  The unstable political environment led to several setbacks, but more recent evaluations point to continuous improvements in service delivery. 
In Pakistan, starting in 2001, the Access to Justice Program (AJP) was supported by the Asian Development Bank (ADB).  The program consisted of a US$330 million loan; a $20 million Technical Assistance loan; and a $25 million endowment fund called the Access to Justice Development Fund (AJDF). The AJP includes a wide range of reforms. It has components on caseload management, legal empowerment, reforming the prosecution service, administrative justice, fiscal reforms in the judiciary, and police reform. The program was initially designed for a five year period and then extended to mid-2008 due to insufficient capacities to absorb assistance at this high level. By that time, only about half of the Technical Assistance loan had been spent. The ADB refused the request to again extend the Technical Assistance loan until 2011, which focused on the more tangible capacity building components. The rest of the loan, which largely focuses on construction, was extended. Today there are mixed reports about the success and sustainability of AJP reforms.  There are indications that targeted projects have decreased backlogs in selected courts and there was a quite positive impact on the overall business climate. AJP has also increased the budget outlays devoted to the judiciary by an average of 30%. At the same time very critical reports of justice sector performance in other parts of Pakistan are emerging. It is important to note, that similar difficulties were experienced as during the implementation of the LJSP. These included limited absorption capacities for significant capacity building, and difficulties with establishing meaningful and practical M&E frameworks. At the same time, like the program in Uganda, the effort in Pakistan focused quickly on reforms that would make a difference in service delivery across the country, and results were visible relatively early and continue.  Creating visible improvements in service delivery is extremely important for gaining broader support for ongoing reforms, including support from the general public.
2.2	Relevance
This section assesses whether the LJSP contributed to higher level objectives of the aid program (outlined in country and thematic strategies), and specifically whether LJSP’s objectives are relevant to GoA and GoPNG priorities, and the context/needs of beneficiaries.
Against a backdrop of project aid to various PNG law and justice agencies between 1988 and 2005, the LJSWG under the auspices of the DNPM commenced work in 1997 to confirm and consolidate common issues as a basis for formulating PNG’s National Law and Justice Policy and Plan, which was adopted and endorsed by the National Executive Council (NEC) in August 2000. In 2007 the NEC endorsed a White Paper on Law and Justice endorsing the policy and the Sector Strategic Framework (SSF).  The fact that the sectoral approach is a home-grown initiative is probably the single most important factor in assessing the LJSP as highly relevant. It also accounts for the sector’s uniqueness in attempting to define and implement the meaning of a “sector wide approach” as one in which most of the key formal justice sector agencies are aiming to engage as one entity.  
At the time of the design of LJSP in 2003 there was no approved development cooperation strategy for the Australian Aid Program in PNG. The LJSP guiding principles, developed jointly with the LJSWG in 2003, helped to ensure the program’s relevance to GoA and GoPNG priorities. These principles include (amongst others) that the program:
•	Actively support the strategies, objectives and priorities established by the GoPNG and the NCM to plan and administer an appropriate and affordable law and justice system for PNG; and
•	Be jointly prioritised and coordinated between the GoPNG and Australia in an on-going partnership to drive significant improvements in the justice, law and order situation.
A Joint Aid Review in 2004 led to the Papua New Guinea-Australia Development Cooperation Strategy 2006-2010, through which Australia aims to help the PNG Government implement its own medium term development goals. The law and justice sector forms one of seven expenditure priorities in the GoPNG’s MTDS 2005 – 2010. 
Australian support through LJSP relates directly to two of the Strategy’s four focal areas: improved governance and nation building; and improved service delivery and stability. It also relates indirectly to the other two: generating broad-based growth and tackling HIV/AIDS, as well as to Australia’s Gender Equality policy. 
In August 2008 PNG and Australia signed the PNG-Australia Partnership for Development, heralding a new era of development cooperation that aims to accelerate progress towards the Millennium Development Goals (MDGs) and PNG’s other development priorities. Schedules for these first five priority outcomes were endorsed at the 19th Australia - Papua New Guinea Ministerial Forum in Brisbane on 10 June 2009. The Forum also announced three new priority outcomes, one of which is law and justice. A Schedule for this was to be considered at the 2010 Ministerial Forum.
The flexibility of the LJSP and the redefinition of its purpose in 2007 had enabled it to maintain its close alignment with and relevance to the higher level objectives of the aid program and the priorities of both GoA and GoPNG.  
The LJSP reflects the broad international consensus that aid is more effective when it is consistent with partner government systems and policies. This forms one of the basic pillars of the 2005 Paris Declaration on Aid Effectiveness. Significantly, LJSP was a leader in ensuring that all activities funded through the program are prioritised by the LJSWG and channeled through the PNG Development Budget (DB) process, for approval by the NCM.  This gives new meaning to the term “use of government systems.”  
The context and needs of the beneficiaries were clearly recognised in the objectives as expressed through the three pillars of the PNG National Law and Justice Policy and the five Outcomes of the SSF.  In practice however, driven by the focus of the sector at the time, the LJSP placed strong emphasis on improving the work of the national level agencies within the sector, resulting in less attention to promoting restorative justice and sub-national service delivery.  
LJSP complied with several of the broad intents of the Paris Declaration and Accra Agenda for Action, in particular those relating to ownership and alignment.  These are reviewed in more detail in Annex 5. LJSP was consistent with the intent of harmonisation, although the need for harmonisation was much less in this sector compared to others, as AusAID was by far the main donor to the justice sector in PNG. Evidence of compliance with Managing for Results and the applicability at the program level of mutual accountability was less clear. 
In summary, the LJSP was highly relevant to higher level objectives of the aid program and to GoA and GoPNG priorities, whilst somewhat less relevant to the context and needs of end-users of law and justice system.
2.3	Effectiveness
In reviewing the effectiveness of LJSP a distinction is made between a) achieving the overall objective of a sector wide approach and b) achieving sector performance related to the goals of the Sector Strategic Framework (SSF). This section assesses the effectiveness of LJSP in achieving the overall goal of creating a sector approach and in achieving the PNG sector goals.  The effectiveness of LJSP relative to the SSF goals is discussed in Annex 6.
LJSP effectiveness in achieving the overall goal of creating a sector approach
The program was very successful in facilitating the development of a good understanding of the meaning and importance of taking a sector wide view and approach to justice sector operations, management and policy.  This was evident from reviewing the relevant documents as well as the interviews with key PNG counterparts, former and current program staff and others. The recognition of the importance of a sector wide approach had predated the program, was enshrined in the SSF and was the basis for structuring the program design. Agencies had been operating and planning very much as separate entities when the program started.  By the end of the program there remained areas to improve, but the coordination structures created to develop a sector approach, i.e. the NCM, the LJSWG, and the Activity Management Teams (AMTs), had become relatively stable institutions.  The focus on combined planning and policy development across all agencies was well established for development activities and the development budget (DB).  All justice agencies applied the same planning and budgeting processes and cycles, and used largely the same PNG systems. Their corporate plans were linked to the same goals and guiding principles, further solidifying the continuity of a sector wide approach. These are areas that the program supported effectively. 
Other important sector wide functions were less solidly integrated. There was still the key issue of who was responsible for the work of the Law and Justice Sector Secretariat (LJSS) and where it should and may reside in the future. At the end of the program the LJSS continued to be fully funded out of the DB and it appeared to be very much under the control of the program.  Similarly, the good work of the the Community Justice Liaison Unit (CJLU) had ad hoc project-based links to, but was not part of, on-going sector agency activities, did not fit well within the SSF framework and required program funding for all of its activities.  Furthermore, in terms of core business activities and recurrent budget planning, the agencies still had a way to go to also apply a sector wide view and approach. Interviewees representing sector agencies and institutions recognised that these were key issues to be tackled and needed to be addressed in subsequent years.
LJSP effectiveness in achieving the PNG Sector Goals
The National Law and Justice Policy and Plan of Action (NLJP) predated the LJSP. It represented a significant local initiative that was unique because it recognised not only the need for a sector wide approach but also the importance of taking a restorative justice approach. Internationally the importance of taking a restorative justice approach is increasingly being recognised and is considered best practice. It is particularly relevant in a country like PNG. The resolution of conflict in traditional PNG societies had long relied on principles of mediation, restorative justice and reciprocity.
It is surprising then that the program focused strongly on strengthening largely formal and national level institutions before engaging at the local level and with traditional leaders, NGOs and CSOs.  Equally surprising is its primary concentration on criminal justice sector improvements.  Strictly speaking only goal 1 (Improved Policing, Safety and Crime Prevention) and goal 3 (Improved Reconciliation, Reintegration and Deterrence) relate mainly to criminal justice matters. Civil and administrative justice matters and agencies were captured under goals 2 (Improved Access to Justice and Just Results), 4 (Improved Accountability and Reduced Corruption), and 5 (Improved Ability to Provide Law and Justice Services). They were not regularly pursued under most LJSP program plans or activities.  The significant crime and safety issues faced across PNG justified the focus on criminal justice matters. In reality, civil matters such as access to and use of land, water and other natural resources, property and inheritance rights are not only more important to many people but also often at the heart of violent conflicts and crimes committed.  
Also, considering that 85% of PNG’s population lives in rural areas, the program’s overall effectiveness across the country was limited by its strong focus on the National Capital District (NCD) and the national agencies located there. Activities focusing on systematically enhancing local level justice sector activities outside of the NCD such as the village courts, the model court operations in Lae and Mt. Hagen, establishment of legal aid branch offices and legal aid desks became effective largely only during the last year of the program. 
As outlined in more detail in Annex 6, effectiveness across the five program goals was mixed. The very uneven allocation of funds to the five goals reflects variation in attention to the different goals: “45% of the DB was allocated to Goal 5 and 29% to Goal 1. Goal 4 (accountability and reduced corruption) received only 5% of the DB.”   
2.4	Efficiency
This section assesses the efficiency of the implementation of the LJSP, including the performance of the contractor in managing the LJSP in meeting contract service delivery objectives as listed in Attachment A to the TOR.
When LJSP was designed, sectoral approaches were relatively new, and this particular approach was highly innovative.  In this situation it is difficult to find suitable comparators, as illustrated in Section 2.1. Optimal value for money was recognised but did not feature significantly in the design. 
Effectiveness in use of time and resources to achieve the outcomes: The main documentary sources for assessing the performance of the contractor in managing the LJSP activity each use different analytic categories and/or indicators, and none align directly with the Contract Service Obligations as contained in Attachment A to the TOR.  Overall, however, these sources rated most aspects of contractor performance in most years as satisfactory.  
This masks the program’s considerable start-up difficulties. These difficulties were associated with lack of understanding of the paradigm shift that the sector wide approach represented. For the first two years significant shortcomings were experienced in: hiring the right people; setting up adequate management structures; coordination among advisers and AMTs; reporting structures that fulfilled contract requirements but were not aligned to agency and sector needs; communication issues across staff, AMTs and getting the right information to the LJSWG and NCM; capturing lessons and learning; and absence of a meaningful M&E plan and baseline data.
Slow progress in implementation led to tension with the emphasis in the Head Contract on expenditure, and ACIL’s inflexible interpretation of the Head Contract in the early years.  A key issue that affected performance was the visible and divisive tension between the Team Leader and the Strategic Policy and Evaluation Director. There was delay in the recruitment and mobilisation of a replacement Team Leader, and the replacement proved unsuitable. There was an absence of strategic thinking and a generally poor relationship with the LJSWG at that stage. The contractor’s use of the staff performance appraisal system documented in its Program Administration Manual and the timely mobilisation of approved personnel were rated as unsatisfactory.  Requests from AusAID for information relating to the program were responded to but in many instances not promptly. This was an ongoing issue, about which AusAID had expressed serious concerns. The LJSWG was insufficiently involved in the development of the Annual Program Plan for 2005. This created substantial concerns amongst LJSWG members. Progress on gender equality was hindered by lack of development of the proposed strategic framework during 2004, omission of gender issues in the development and vetting of proposals, and the lack of a dedicated capacity within the LJSP.
These early management issues seriously impeded program progress to the extent that changes were achieved late and impact was not experienced or visible until late in the program. The contractor took a number of steps to address these problems, particularly since 2007. Defining and implementing the needed changes came quite late. The addition of highly qualified local advisers in combination with international advisers with solid experience in core agency operations related to service delivery made a difference in the last two years. The roles and responsibilities of advisers that were not embedded in an agency (i.e. the Core Support Team (CST)) were not well communicated to agencies.  
Risk management and anti-corruption: The contractor put in place strong financial and anti-fraud systems.  However, there were still issues to be dealt with, particularly relating to apparent corruption issues in procurement of construction projects.  While the program had solid control and audit processes in place, it had little recourse if corruption was suspected during the tendering process. Risks to achievement of objectives are discussed in Annex 7. 
In summary, while the efficiency of the program is assessed as satisfactory in later years, it cannot be rated as fully adequate given the deficiencies in the earlier years of the program.
2.5	 Impact and sustainability
Impact
This section assesses the impact of the LJSP, distinguishing between impact on sector performance, on end users, and impacts from external factors. Lack of baseline data contributes to the difficulty of distinguishing program impact from that of the sector, agency and other factors.
Impact on sector performance: Impact is seen in great ownership and support for a sector wide approach across all agencies.  While the recognition of the importance and benefit of taking a sector wide approach predated the program, all of the sector coordination, planning and administrative support functions were created and maintained with program funds and support.  More importantly, today’s sector-wide planning would have been difficult to achieve without the program’s support. The sector made great strides in evolving into a mechanism for combined planning and budgeting across agencies at least related to DB activities, such that all development funding, sourced both from GoPNG and Australia, was managed and administered through the sector.  A limitation is that recurrent funding was not subject to sector consultations. Furthermore other donor funding was not always part of the sector planning process. In several justice sector agencies, such as the Magisterial Services (MS), the OC and Public Solicitor (PS), these important corporate functions (i.e., basic HR systems and functions, budget and finance and IT as well as overall agency planning) were solid by program end. In others, such as the RPNGC, they were well functioning but still fragile, depending greatly on a few staff in finance and HR positions without being fully supported and institutionalised.  
Impact on end users: Some agencies, especially the OC, the MS, and the Office of the PS had progressed to the point that their improved work and service delivery is visible to the end user.  The OC had been increasingly successful in pursuing complaints filed. The significant improvements in case management in magistrate registries had resulted in reduced backlog and speedier processing of cases. The delivery of legal aid by the PP had vastly increased across the country. Support to the Office of the Solicitor General (SG) resulted in solid, sustainable improvements that had already saved the government significant money. Juvenile justice reforms resulted in the creation of specialised juvenile courts, juvenile reception centres and increased application of mediation and diversion.  This had led to a 62% reduction of juveniles held in remand and a 27% reduction of youth sentenced to detention.  Anecdotal evidence indicated that strengthening village courts across the country had led to positive outcomes for end users, including women and girls, but these processes were still evolving.  Progress had also been made in stepping up awareness of family violence issues and in providing protection for victims of family violence in Port Moresby, Lae, Mount Hagen, Madang, Rabaul, and Goroka.  The Yumi Lukautim Mosbi (YLM) project, while pre-dating the program, gained significant momentum after being supported by the program. It had resulted in significant improvements on safety and security in Port Moresby, and in linking juvenile offenders to training jobs. Most of the visible results for end users had only materialised in the last year.  There was currently good momentum for making greater strides in achieving visible results and greater impact on justice sector end users.  
Impacts from external factors: The program’s impact was limited by its inability to address a number of external initiatives.  These are discussed in Annex 8.
Sustainability
This section assesses the sustainability of any improvements in sector performance induced by the LJSP and the factors critical to the continuity of any positive trends in sector performance, and specifically: 
•	Whether PNG stakeholders have sufficient ownership to maintain LJSP’s outcomes after GoA funding has ceased;  
•	Whether there are any areas of LJSP that are clearly not sustainable. 
The most important factor likely to ensure sustainability in improvements in sector performance is the strong level of ownership across justice sector agencies and strong support for a sector wide approach.  It remains to be seen, however, whether the relatively involved sector coordination structure will continue to be seen as valuable and if it can be sustained with government funding in the long term.  This would require inclusion in the budget of DNPM or the Ministry of Justice, or the creation of a separate budget line in the government’s budget for a sector entity. Without this the current structure is not sustainable. The emerging pattern of significant under funding of the GoPNG contribution to the LJS DB’s central imprest account as occurred in 2007, 2008 and 2009 could undermine gains achieved so far. The program shows an awareness of its reliance on favourable political factors being in play and has taken steps to build on this to increase its sustainability.
The program’s emphasis on establishing core corporate systems and capacities in all sector agencies had created dedicated staff and even specialised units supported by finance and HR systems that were used government-wide. There were no indications that these efforts would not be sustainable, although staff turn-over in these positions posed some challenges, especially within the RPNGC. Two additional strategies recommended to be undertaken by the program would enhance the long-term sustainability: the creation of a pool of local trainers that could assist across all agencies; and (where they had not already been developed) the development of practical operations manuals. 
Case management improvements in MS, and the expansion of the services of the Offices of the PS were well on their way to being institutionalised. Similarly, the significant outside funding for YLM activities indicated sustainability, even expansion opportunities.  Overall, the program could have benefited from the sector strengthening its focus on sustainability planning.  This was being started in relation to facilities management but a range of other sustainability planning options could have been introduced.  There seemed to be little emphasis on up-front sustainability planning throughout the program.  For example, the reduction of juveniles held in remand and reduction of youth sentenced to detention  should have created significant savings in the CS.  These savings could be used to sustain the important alternatives created.  There is no indication that the program supported the sector in gathering information about such cost savings – or eventually developing data collection schemes that indicate if the alternatives created resulted in other benefits, such as keeping juveniles from re-offending.  Both would be important information for the sector to use in budget negotiations with the government in support of funding for these activities in the future.
The sector was currently exploring sustainability options for the CJLU. Although the CJLU had supported many important activities at the community level, its operations were not well connected or integrated with other sector priorities and individual agency functions. Some of the work of the CJLU had included strengthening NGO’s finance, HR, procurement, and other administrative systems. This may have been important for ensuring alternative service delivery in locations where well run NGOs were scarce but may not have been the best use of limited funds. Linking the sector and its agencies to various resources at the community level would increase in importance as the sector focuses in future on service delivery in all provinces. Unless the CJLU could demonstrate cost savings and other benefits for sector agencies it was not likely to be sustainable in the long run. The current situation where the CJLU received sector funding and then funded NGOs to train community members or conduct activities created additional bureaucratic layers and cost. According to the 2007 annual report, 43% of the CLJU budget was spent on its own unit development and management.
The sector still lacked human capacities in many areas. The sector and most agencies could draw upon very well qualified people in Port Moresby, but were thinly stretched at the management level. Lack of human resources would be an even greater challenge when reforms were to be expanded to the provinces and districts. Improvements in the formal sector might not be a realistic solution for many years given the limited reach of the sector and the significant resource needs over time.
In summary, capacities to sustain core corporate functions are expected to be sustainable in most agencies; sustainability in relation to broader program goals is questionable. All major sector operations, especially the LJSS and the CJLU continue to be fully funded by the program and would not be sustainable if funding ceased.
2.6	Gender Equality
This section assesses whether the LJSP advanced gender equality and whether the program:
•	Promoted more equal access by women and men to the benefits of the LJSP, and more broadly to resources, services and skills;
•	Promoted equality of decision-making between women and men;
•	Helped to promote women’s rights;
•	Helped to develop capacity (donors, partner government, civil society etc.) to understand and promote gender equality.
It also assesses the extent to which gender equality and HIV/AIDS mainstreaming has been integrated into the LJSP.
Progress in advancing gender equality and in mainstreaming gender equality and HIV/AIDS in the LJSP needs to be assessed within the PNG context. In this context not only are the sector and the society strongly male dominated, but the sector’s agenda determines what initiatives are to be prioritised and resourced through the DB.  By 2005 gender had not been identified by the sector as a priority. Gender was absent from the Sector Strategic Framework, and – significantly – from the Performance Monitoring Framework (PMF) developed for the sector by the JAG.  
A thorough gender analysis was undertaken as part of the program’s design.  It covered program context and points of entry, gender analysis of LJS agencies, and major law and order issues affecting women and children. This analysis was important in providing LJSP with a platform from which to advocate successfully for the development of a gender strategy.  
By October 2005 a full-time national gender adviser had been appointed, and the LJSP had detailed its support to the sector in 2006 in improving gender equality. This included four broad areas: Support for implementation of the Gender Strategy in LJS; Corporate Planning and Coordination; Agency Level; and LJSP and LJSS Roles.  Marketing the strategy to the sector was challenging. Champions were sought initially within LJSP management and advisers, and then amongst the sector’s leadership.  With the introduction of Activity Management Teams (AMT) in 2006, the gender advisers worked with the all-male HR AMT, who were hostile initially. Recognition by the sector of the importance of gender and HIV/AIDS was given a major boost by pressure from the Family and Sexual Violence (FSV) Action Committee to become actively aware and find solutions.  The issue of FSV is credited with having facilitated some of the first sector coordination on cross-cutting issues. Today “there is positive support and commitment from the leaders of the sector and….HIV/AIDS is seen as everyone’s concern and core business of the sector and as contributing to gender equality.” This is a significant achievement. The formation of a Cross Cutting Issues Team played a key role in gender mainstreaming. It improved coordination between agencies and enabled all agencies to participate in conducting Gender, EEO, and HIV/AIDS and FSV programs. This contributed to more equal access to the benefits of the LJSP by women and men employed in the agencies, and helped to promote the rights of women employees.  Agency workplace policies on Equal Employment and HIV/AIDS were developed in 2008, and their implementation is expected to be emphasised in the PALJP.  
LJSP support contributed to progress made in promoting more equal access by women and men to services through the efforts of the Director of the Village Courts Secretariat to have every province recruit at least one female magistrate per court. All Secretariat village court staff are required to undergo training on human rights and gender.  The building of new infrastructure in parts of the country to cater for the needs of women and children was expected to yield future benefits.  The number of female magistrates in PNG rose from about 10 in 2004 to about 350 in October 2008. More broadly, however, limited impact had so far been achieved in terms of more equal access to resources, services and skills at a subnational level, although some employment for women and youth had been created in NCD through YLM.
A significant contribution was made to developing capacity to understand and promote gender equality and to mainstream HIV/AIDS among advisers, through advisers in the sectoral agencies, and in civil society through YLM. Together with BEDP personnel, LJSP advisers extended capacity development beyond the sector by facilitating training for 90 Advisory Support Facility Advisers on gender in PNG cultural context. Compared to some other AusAID supported programs in PNG, LJSP provided a stronger level of resourcing for gender. By 2007 a full-time national HIV/AIDS adviser had been appointed, as well as a second gender adviser.
LJSP made only very limited progress in promoting equality of decision-making between women and men in the justice sector. The Team noted, however, with interest the LJSP’s role near the end of the program in supporting the National Council of Women to obtain funding from the UN Democratic Electoral Lobby Fund to provision of support to women’s representatives to local level government, provincial assemblies and urban authorities.  This is a significant development. 
Within the PNG context, although there is little visible impact so far, LJSP is assessed overall as having laid the groundwork and made a significant contribution to gender mainstreaming and advancing gender equality in PNG, except in promoting equality of decision-making between women and men.  
2.7	Monitoring and Evaluation
This section assesses whether the LJSP’s M&E system effectively measured progress towards meeting objectives, and specifically whether
•	Evidence exists to show that objectives have been achieved;
•	There are features of the M&E system that represent good practice and that improved the quality of the evidence available;
•	Gender-disaggregated data were used to measure the outcomes of LJSP on men, women, boys and girls;
•	LJSP’s M&E system collected useful information on cross-cutting issues.
M&E was a vexed issue throughout the program.  This is illustrated by a chronological summary of M&E of LJSP contained in Annex 9, derived from Six Monthly Reports, Annual Reports, Annual Plans and other documents. It took about four years of implementation to reach a clear understanding between key stakeholders that progress towards meeting the SSF objectives needed to be measured at three distinct levels:  sector, agency and program, and that responsibility for these levels of M&E rested with different parties.  Sector outcomes were measured through the Program Monitoring Framework (PMF) and the Annual Sector Performance Reports produced by the JAG with some contribution from the LJSS. Agency objectives were captured (to a limited extent, with LJSP support) by the agencies. The program itself was assessed at two levels: strategic management performance – against program governance and strategic management indicators; and activity implementation – through the reports of program advisers. This understanding was captured in the LJSP M&E Framework (MEF) dated January 2008, for use during the remaining 15 months of the program.  This slow progress was attributed to a number of reasons, including a protracted design stage, disagreement over resourcing, a lack of coordination between the managing contractors of the LJSP and the JAG, and the limited capacity of GoPNG in M&E. The role of the JAG in M&E and its creation before and outside of the program led to confusion and disagreements about the respective responsibilities of the JAG and LJSP for M&E.  This deflected from the important issue of designing an effective M&E system and establishing a baseline. 
Evidence to show that objectives were achieved is limited by the framing of the goals and outcomes of the SSF in terms that are not measurable. This problem has been partially addressed by the development (late in LJSP) of Key Result Areas, and recent reports against these.  No baseline was undertaken at the start of the program, so there is little robust evidence to show the extent of improvement.  Evidence is largely anecdotal. 
Some features of the M&E system documented at January 2008 reflect good practice, but were developed extremely late in the program. For instance, there was some attempt through adviser support to develop M&E data and capacity at the Agency level. The MEF clarified what information was to be collected, when, where from and how, and adequately addressed all levels in the design logic.  Responsibilities were clarified. Anecdotal evidence suggests that agencies were beginning to understand the value of data for their own operations and were showing a greater interest and willingness to collect and use it. It is not clear, however, how widespread or evenly this capacity and understanding was distributed.  
At the sector level, data in the Annual Sector Performance Reports are inconsistent from year to year and no consolidated or trend analyses from these annual reports were made.  The Information for Monitoring, Evaluation and Reporting Project Formulation Document (PFD) which focused on the enhancement of M&E in Agencies, itself provided no concrete M&E information at all about progress and achievement over the past three years The LJSWG and the NCM have not demanded or used data or analysis of the quarterly PFD Reports against the AMT reports, and have not critically evaluated progress of these projects. This suggests a lack of M&E capacity at the Sector level. There seems to be little if any ownership of the PMF and the Annual Sector Performance Reports by the NCM and LJSWG.  
The capacity of the M&E system to measure the outcomes of LJSP on men, women, boys and girls – i.e. the end user - is extremely disappointing.  Agency data collection frequently does include data on gender, but this is not recognised and collected systematically. Partly this is because there was little program focus on end users of the law and justice system.  However it is disconcerting to note that the 61 indicators in the PMF developed by the JAG included none that were gender disaggregated.  Five years of gender disaggregated performance data have thus been lost.  Furthermore this is also true of the 17 key performance indicators of the Sector PMF included in the M&E Framework of January 2008. It is even difficult to assess what gender impact was experienced by those who were directly involved in the program. The contractor’s own reporting in gender disaggregated form was very limited.  Much more could have been done, for instance, to report the gender composition of groups that were trained or participated in capacity development activities.  This suggests that advisers were not required to report routinely on gender participation in capacity building activities. 
The M&E system did, however, report usefully on cross-cutting issues, e.g. on activities that were gender focused, such as FSV, and on HIV/AIDS initiatives.  The program could have done more to capture and evaluate many of the significant gender implications of activities such as YLM and the work of the CJLU that were not specifically gender focused. 
LJSP gave little priority to its M&E system and overall the result was much less than satisfactory, except in the final 12 – 18 months of the program. 
2.8	Analysis and Learning
This section assesses whether the LJSP was based on sound technical analysis and continuous learning, and specifically:
•	How well the design was based on previous learning and analysis; and
•	How well learning from implementation and previous reviews (self-assessment and independent) was integrated into the activity.
The outcomes in LJSP’s initial design reflect the three pillars of the National Policy. The design was modified in 2006 to link its purpose to the five outcomes of the SSF. Whilst this change made good sense, the Agency Corporate and Annual Plans do not link to the SSF. The purpose was redefined again in 2007 to emphasise activity planning, designing, implementing and reporting related to the National Policy and the SFF. LJSP’s design was thus evolving over the years.
A suite of analyses was undertaken by the program to inform the initial design, and form Annexures to the Program Design Document. These appear technically sound, although one (Fraud and Anti-Corruption Options Paper) was based in part on what proved to be a faulty assumption: that the Enhanced Cooperation Program would make a significant contribution. The Gender Analysis and Case Studies provided a strong foundation for the development of a gender strategy and a good performance overall in advancing gender equality. The Expenditure Analysis contained a description of the PNG Budget Institutions and Systems. The mainstreaming of the DB process in the program clearly reflected understanding of the PNG planning and budget cycle.  Elements of the Incentives Options analysis are recognisable in the competitive nature of the selection of PFDs, and in the competitive grants element of the CJLU. The Fraud and Anti-Corruption Options Paper referred to learning derived from the RPNGCDP III, the OC Institutional Strengthening Project and the PNG Incentive Fund. The problem analysis acknowledged that analysis should be “an ongoing process to refine and re-shape the program’s direction and activities throughout implementation,” and that “full implementation of the sector-wide approach requires that appropriate mechanisms are also established at provincial and local levels.” It highlighted economics, poverty and waste; poor system performance; and poorly defined institutions and lack of community influence, discussing these in relation to the three pillars of the National Policy.  
A great many independent reviews were made of aspects of LJSP, as well as some commissioned by the contractor itself. The lack of an adequate M&E system makes it difficult to track (except in general terms) how well this learning was applied. The evolution of the design suggests learning from implementation was applied, although learning and application seem to have been less than systematic. As emphasis grew on planning, designing, implementing and reporting on activities related to the National Policy and the SFF, the emphasis on the third pillar of the National Policy relating to restorative justice appears to have reduced, notwithstanding the initial analysis that highlighted its importance.  At an NCM retreat the Chief Magistrate recognised that the “DB process is more focused on institutional strengthening of agencies, and less on the development of broader law and justice issues such as ‘restorative justice’.” This suggests that whilst continuous learning around the planning and budgeting process took place, sight may have been lost of some of the broader technical and policy issues (including subnational service delivery and restorative justice) that were the subject of early analysis. Indeed there is evidence of lessons that should have been learned from the 2006 DB process that were not applied to improve the 2007 DB process. 
Good modelling of analysis and learning by the Contractor could have played an important role in developing a culture of analysis and learning in the program and the sector. There is some evidence however that the Contractor did not maximise its learning from its previous experience in the sector. Prior to the program ACIL (since subsumed into Cardno ACIL) had extensive experience in the sector of over a decade.  Some of the advisers novated from those projects brought a project delivery rather than a capacity development mentality to LJSP. The review of the program undertaken by Cardno ACIL led to significant improvement in the program from about 2005 onwards. This is clear evidence of learning, but was very slow in coming. In the first year or so the Contractor did not seem to have systems and processes in place to evaluate and learn from what it was doing. 
On the positive side, AMTs contributed well to organisational learning and sharing across the sector, and should be commended. Overall however there was a conservative approach by the sector and the program to learning from pilots. An insightful analysis was made in 2006 of the three provincial engagement pilots, and lessons identified to guide the way forward. The evaluation of these same pilots again in 2009 suggests that learning from the previous analysis was not applied.  Some of the successes of YLM have been visible for some time.  Much more could have been learned from it than has so far been the case.  The sector could use a range of media, as YLM has done, to raise awareness of the work of the sector. For example, media interviews in Tok Pisin with some of the women trained as village court magistrates could be used to improve community understanding of the role of these magistrates.    
Although the program design was based on previous learning and analysis, integration into the activity of learning from implementation and previous reviews was slow and tended to lack a systematic approach. If the original design for this very innovative program had adopted an explicit action learning approach, with much shorter cycles, this could have greatly facilitated continuous learning and could have contributed to timelier implementation.
3.	Evaluation Criteria Ratings

Evaluation Criteria	Rating (1-6)	Explanation	
Relevance	5	LJSP was highly relevant to higher level objectives of the aid program and to GoA and GoPNG priorities, but somewhat less relevant to the context and needs of end-users of law and justice system.	
Effectiveness	4	Overall, although effectiveness across the five program goals was mixed, the program was very successful in facilitating the development coordination structures and translating the meaning of a sector wide approach to agency planning, operations, management and policy.  	
Efficiency	3	While the efficiency of the program is assessed as satisfactory in later years, it cannot be rated as fully adequate given the deficiencies in the earlier years of the program.	
Sustainability	4	Capacities to sustain core corporate functions are expected to be sustainable in most agencies; sustainability in relation to broader program goals is questionable. All major sector operations, especially the secretariat and the CJLU, continue to be fully funded by the program and would not sustainable if funding ceased.	
Gender Equality	4	Within the PNG context, LJSP is assessed as having laid the groundwork and made a significant contribution to gender mainstreaming and advancing gender equality in PNG, except in promoting equality of decision-making between women and men.  	
Monitoring & Evaluation	2	LJSP gave little priority to its M&E system and overall the result was much less than satisfactory, except in the final 12 – 18 months of the program.	
Analysis & Learning	3	Although the design was based on previous learning and analysis, integration into the activity of learning from implementation and previous reviews was slow and tended to lack a systematic approach.	
Rating scale:
Satisfactory	Less that satisfactory	
6	Very high quality	3	Less than adequate quality	
5	Good quality	2	Poor quality	
4	Adequate quality	1	Very poor quality	

4.	Lessons Learned and Recommendations
A great many reviews and evaluations of the LJSP have been undertaken and most of these have identified lessons learned.  Although many of those lessons deserve to be reiterated, this section focuses on those lessons that are of most relevance to (a) PALJP, (b) to other future support to the Sector; and (c) to other future support more broadly, e.g. to SWAps in other sectors, not necessarily limited to PNG.  The recommendations are framed with this in mind.
Lesson 1 Service delivery focus (Section 2.5): A sector that comprises multiple national government agencies is inherently complex.  It has at least two major challenges: (a) to develop a way of just operating as a sector – not even delivering better services as a sector - and sufficient agency capacity for that; and (b) to improve services delivery, especially at sub-national levels.  Developing a way of operating as a sector and the necessary capacity can be a lengthy process.  Based on LJSP experience this can be of the order of five years – even when key agencies recognised the importance of a sector wide approach before the program started. This long gestation period requires a high level of risk tolerance for both donors and contractors and can lead to delay in improving service delivery. Strengthening core corporate capacities in justice sector agencies, the way it has been stressed in the LJSP, is important to enable agencies to develop internal capacities to support delivery of quality services in the future.  However, actually delivering good services is the ultimate goal and requires different capacity building activities and relevant frontline staff and systems to assist them. The agencies that made the greatest progress were those that had both good leadership and capacity building support both for corporate capacities and for strengthening front line service delivery. Examples from other sector wide justice system reforms have also shown that more visible impacts can be obtained by providing support for strengthening service delivery capacities at the same time as support for corporate capacities. This also produces more relevant engagement at each agency and across the sector.  This is because corporate functions are limited to a small number of “administrative” staff, important functions that are marginal in comparison to the larger number of staff supporting core agency line activities.   
Recommendation 1: 
(a)	That the PALJP ISP engages with the sector and agencies with a view to undertaking solid needs assessments and in refocusing work plans on increasing service delivery capacities at both the national and sub-national levels;
(b)	In the design of future programs of support, whether for the law and justice sector or for other sectors, AusAID and partner stakeholders take care to ensure that support for strengthening service delivery capacities is given equal attention to support for corporate capacity development.
Lesson 2 Common understanding of roles and capacities (Section 2.2): Applying a sector wide approach to justice system reform requires that all involved (each agency, the implementing contractor and the funding agency) develop a shared and clear understanding of what this means for their respective roles and responsibilities. The roles and responsibilities of each agency, the contractor and funding agency are likely to shift over time as capacities develop and needs change. Both the sector and the Implementing Service Provider (ISP) should take stock regularly of the level of engagement needed and available capacities.  The ISP needs to be flexible in providing different levels of support, frequently being in a facilitator role. This requires that advisers not only have relevant substantive expertise but experience in change management processes, knowledge of international comparative experiences with which to link, and are creative in supporting experimentation and learning. 
Recommendation 2: That the NCM, with support from the PALJP, consider
(a)	Undertaking periodic stocktakes to determine whether agencies have the capacities to participate in sector wide activities, to take the lead in formulating policies and implementation plans and in managing change processes; and 
(b)	Using these stocktakes to guide agencies in developing specific initiatives for addressing capacity deficits and identify roles and responsibilities, for funding through the DB.    
Recommendation 3: That the PALJP ISP and AusAID engage in sufficient analytical work to predict change patterns and adjust to changing needs with flexibility.
Lesson 3 Use of government systems (Section 2.2): The use of the GoPNG DB as the path through which all program activities are identified and selected for funding has proven an effective mechanism for operationalising the use of government systems in line with the Paris Declaration and the Accra Accord. It has also proven effective for ensuring cooperation between the government agencies in the sector. The DB process does not, however, require agencies to review and reallocate resources within their recurrent budget.  This approach to using government systems combined with the programs focus on reform of corporate functions therefore tends to encourage agencies to view change as applying only to their administrative rather than their core service delivery activities.
Recommendation 4:  That the NCM and AusAID consider how the PALJP can support an approach to sectoral planning and budgeting that better integrates the development, recurrent and supplementary budgets.
Lesson 4 Communicating the meaning of a sector wide approach (Section 2.1): The meaning and importance of a sector wide approach needs to be communicated clearly to other relevant agencies and the government in general. This is key to ensuring that government planning and budgeting reflects and considers the needs of the sector needs before the needs of individual agencies.
Recommendation 5: That PALJP ISP supports the sector in developing its capacity to collect meaningful information on its needs and its progress, and to develop systematic and ongoing communication mechanisms to ensure government support.
Lesson 5 Constraints to broad-based engagement (Section 2.5):  Justice sector reform needs to be very inclusive, so as to ensure that resources are available and linked to justice sector operations.  The strong restorative justice emphasis of the SSF implies there should be a greater focus on inclusion of civil justice issues that are so important to people throughout PNG. This will become even more important as the sector expands its efforts to reach the provinces and districts. Other non-justice sector agencies, civil society and communities need to be involved.  The program has experienced difficulty in finding entry points for civil society engagement.  The most fruitful entry points have been found to be specific issues, as these are likely to be catalysts for policy dialogue between the sector and civil society.  The use of the GoPNG DB has proven not to be an effective mechanism for ensuring that civil society organisations have a place in the decision-making process that leads to the identification of activities for funding from donor sources. This suggests that there is a need for the NCM and AusAID to consider alternative ways to channel funding for community-based approaches to restorative justice. It is recognised that AusAID’s Democratic Governance Program (DGP) was established to support the building of such links and provide funding directly to CSOs. However, the DGP potentially serves many other sectors and is vulnerable to budget cuts as experienced in 2009. Given the importance of broad-based engagement to the law and justice sector’s service delivery, it seems unwise to rely on the DGP as the main source of support for civil society engagement in the sector.  This comment also applies to other sectors that are heavily reliant on non-government agencies for service delivery.
Recommendation 6: That 
(a)	PALJP ISP engages with sector agencies with a view to supporting them to engage in stakeholder mapping and outreach, and at the same time providing support for partnership development and community engagement capacities;
(b)	PALJP management and advisers be sensitised to look for and recognise emerging issues which may provide entry points for increased civil society engagement, and act promptly to take advantage of the opportunities that such emerging issues may present;
(c)	AusAID considers reserving some of the funding under PALJP to support activities that are both consistent with the National Policy and SSF, but also are not being catered for through the DB process. This could include, in particular, community-based approaches to restorative justice;
(d)	In the design of future sectoral programs (not limited to law and justice) AusAID recognises the risk that reliance on the DB process as the program planning and budgeting  system may result in the neglect of civil society and private sector engagement in the sector, and ensure that future designs include strategies and/or options for mitigating this risk. 
Lesson 6 Monitoring and Evaluation for service improvement (Section 2.7): The highly innovative LJSP presented a wonderful learning opportunity. A high quality M&E system could have both maximised the benefit of that learning opportunity, as well as accelerated the progress of the sector in improving service delivery. If M&E had been seen an essential tool for continuously improving service delivery it would be clear that it was everybody’s business: that of units within agencies, of agencies as a whole, of the sector, of the contractor and of the donor. The design of an M&E system for a sector as a whole, particularly one such as LJS which comprises a multiplicity of national agencies, is inherently complex.  The different levels of M&E need to be identified and the roles and responsibilities for M&E at each of these levels need to be clarified, understood and accepted.  The basic units of analysis at each level and in each agency also need to be specified. For example, at agency level, the target populations for agency services, their gender and demographic characteristics and location need to be specified, as well as the types of services to be delivered and current coverage of target populations.   Baseline data for each unit needs to be collected, and a process developed for sub sequent periodic collections and reports of these.  The capacity in each agency and at each level of the system needs to be assessed, and strengthened where necessary. 
A holistic approach of this type would require much negotiation and substantial resources.  The scale and scope of M&E required by the LJS was seriously underestimated and the respective roles of different stakeholders and their relationships with each other were not negotiated and agreed at the start of the program.  As a result, it was largely left to the contractor to clarify these roles and responsibilities. This finally emerged with the M&E Framework in the fifth year of the program. With the benefit of hindsight, the program would have benefited from a strong early focus on strengthening agencies’ capacities to identify their target beneficiaries, evaluate their service delivery and develop and implement systems for monitoring and reporting meaningfully on their service delivery.  
Recommendation 7: That
(a)	The NCM seek assistance through PALJP for a significant focus on increasing agencies’ capacities to identify their target beneficiaries, distinguish these by gender and age group, evaluate their service delivery and develop and implement systems for monitoring and reporting meaningfully on their service delivery to women and men, girls and boys in the community; 
(b)	The key performance indicators of the Sector PMF included in the M&E Framework of January 2008 should be amended as follows: KPI 1: “The police service meets the expectations of women and men in the community” and KPI 4 “All women and men have greater access to justice services.”  
(c)	The NCM  and AusAID should consider how the PALJP can support refinement of the PMF to ensure that its indicators are gender disaggregated wherever possible, and that indicators are designed to measure the service delivery targets as defined by the agencies;
(d)	The indicators to be used by AusAID for assessing the ISP’s performance include the effectiveness of PALJP development practitioners in increasing agencies’ capacities to identify their target beneficiaries, to distinguish these by gender and age group, to evaluate their service delivery and improve existing systems for monitoring and reporting meaningfully on their service delivery to women, men, girls and boys.
Lesson 7 Gender Equality and HIV/AIDS mainstreaming (Section 2.6): A thorough gender analysis as part the design process is an essential building-block and can provide a crucial platform for subsequently developing and promoting a gender strategy. The recruitment of full time local personnel as gender and HIV/AIDS advisers, the identification of male champions in senior positions, the allocation of adequate resources, and the inclusion of the CCI team in the CST have all played key roles in the mainstreaming of gender equality and HIV/AIDS mainstreaming in LJSP.
Recommendation 8: That
(a)	The NCM and AusAID ensure that PALJP maintains or increases the level of support allocated for gender equality and HIV/AIDS mainstreaming, and continues to support and facilitate the work of the CCI Team;
(b)	The NCM and AusAID use PALJP to support the implementation of the sector’s gender strategy by reviewing and updating the original gender analysis undertaken for the LJSP and using this to drive new and existing initiatives; and 
(c)	As part of the design for any future sectoral program, AusAID requires a thorough gender analysis to be undertaken and documented, linked to baseline data on agency service delivery level, risks, stakeholder analysis, and lessons learned from any relevant previous program.
Lesson 8 Analysis and Learning based on Action Learning Principles (Section 2.7): In any program that is managed by a contractor, the contractor needs to play a leadership role in modelling good practice analysis and learning for the benefit of advisers, the agencies and the sector.  This requires a systematic approach with planning for frequent reflection points and short learning cycles encompassed within larger cycles such as those of Annual Plans and Annual Reports, and Mid Term Reviews.  
Recommendation 9: That 
(a)	The PALJP ISP consider applying an action-learning approach based on adult learning principles to its own operations, as well as modelling them for the benefit of agencies, the sector and the M&E system as a whole;
(b)	Future designs for innovations such as sector programs be based on action-learning principles; and
(c)	In Request for Tender specifications AusAID should require tenderers to show evidence of their understanding of and commitment to systematic action-learning approaches in their proposals.
Lesson 9 Adviser professional development for best practices (Section 2.8): Advisers for a sector wide program need to have good skills in analysing needs and channelling information and support based on priorities set by their counterparts. They need to be facilitators and good communicators – in addition to experts in their field. The successful advisers use their broader knowledge of new developments in their field to connect their counterparts to other international counterparts and knowledge. This is an important contribution to ensuring that the relevant agencies have access to information resources, research and knowledge that enable them to connect to broader knowledge networks and to understand different options for their own operations and inform decisions for further reforms.  This was all left to the initiative of individual consultants and was not part of a more systemic support structure. After over 5 years of operations such an approach has not been systematised. Similarly, the importance for sustainability of creating a pool of local capacity development trainers and training manuals seems to have been overlooked.
Recommendation 10: That 
(a)	The PALJP ISP provides advisers with ongoing assistance to ensure they are provided with best practice information and alternatives from different jurisdictions; and 
(b)	The sector considers seeking support through PALJP for the creation of a pool of local capacity development trainers that could assist across all agencies, and the development of appropriate training manuals.   
Lesson 10 Assessment Centre approach to recruitment (Section 2.7): Good lessons have been learned in developing more participatory recruitment processes in which the local partners have a key role in selecting advisers. This has resulted in recruitment of highly qualified local and international advisers that have a good sense for how to move things forward.
Recommendation 11: That designs and tender specifications for future AusAID programs draw on the strengths of the Assessment Centre approach of LJSP where appropriate.
Lesson 11 Lack of sustainability of LJSS (Section 2.5): The LJSS has evolved into a relatively large bureaucratic structure that consumes more and more resources.  It is not linked to any existing government or sector agency and not likely to be sustainable in its current form.  
Recommendation 12: That the NCM, the DNPM and the Department of Justice and Attorney General consider and resolve the future institutional home of the LJSS, and how it can be budgeted. 
5.	Conclusion
LJSP was a very different and innovative approach to supporting justice sector reform compared to examples in other countries.  Justice sector reform in PNG has been a home-grown initiative and the sector has used support through LJSP to put into practice a mode of operating in which the key formal justice sector agencies engage as a single entity. 
Overall the strengths of LJSP may be summarised as: a high degree of relevance; effectiveness in achieving cooperation of agencies within the sector and sectoral coordination; likely sustainability of the core corporate functions within the sector; and a significant contribution to gender mainstreaming within the sector.  Compared to many other AusAID funded programs, the level of likely sustainability is unusually high. This is largely attributable to the high level of GoPNG ownership of justice sector reforms.  The extent of gender mainstreaming is also unusually high within the PNG context. The main areas where there were significant opportunities for improvement were in the efficiency of the program, analysis and learning, and in particular, in monitoring and evaluation. 
Compared to examples from other countries, there was a much stronger focus on sector coordination and corporate capacity building, and relatively little attention was given to ensuring that service delivery was improved. The PALJP provides a timely opportunity to redress the balance.