Papua New Guinea – Australia Targeted Training Facility (PATTAF)

AidWorks Initiative Number – INI1004

INDEPENDENT COMPLETION REPORT

Colin Reynolds (TL):  Reynolds Consulting
Bridget Laimo: Freelance Consultant
Regan Field: AusAID

September, 2009

Comments to: colin@reynoldsconsulting.com.au or bridget@daltron.com.pg
Aid Activity Summary

Aid Activity Name		
AidWorks initiative number	INI1004  (from October 2008 to March 2010)
INF205  (from April 2002 to Oct 2008)*	
Commencement date	2 April 2002 	Completion date	31 March 2010	
Total Australian $	INI1004 $6.12million  and INF205 $38.8million 	
Total other $	n/a	
Delivery organisation(s)	GRM International Pty Ltd	
Implementing Partner(s)	n/a	
Country/Region	Papua New Guinea	
Primary Sector	Higher education/ economic development	
* Two different AidWorks initiative numbers were created over the life of the initiative. 
Acknowledgments
The team for this ICR consisted of Regan Field, Bridget Laimo and Colin Reynolds.  (Bridget and Colin as independent consultants and Regan as an AusAID representative).  The team would like to thank all respondents who took time to talk with the team, including alumni, their employers, current and past executive members of the Alumni Association, Government of PNG representatives, AusAID officers and Program staff.  The AusAID Minister-Counsellor (Mr Bill Costello) also made time to answer specific queries of the team, which was greatly appreciated.

Also of particular use to the team was a December 2006 draft of a completion report prepared by the senior PATTAF staff.  While this report was subsequently not submitted due to extension of the program, it contained a great deal of useful summary information that would have been difficult to otherwise pull together.  While the ICR team was careful to corroborate this information, the team was very impressed by the frankness and honesty of the discussion provided by this draft, and it is hoped that this candidness would have been retained in any finalised version.  Were more programs willing to provide such honest reporting (thereby giving realistic opportunities for learning), the need for ICRs such as this one may never had arisen.

Author’s Details
Colin Reynolds:  colin@reynoldsconsulting.com.au 
Bridget Laimo:    bridget@daltron.com.pg
Regan Field:      regan.field@ausaid.gov.au 

List of Abbreviations

ADS	Australian Development Scholarships
ALAS	Australian Leadership Award Scholarships
ALO	AusAID Liaison Officer
AMC	Australian Managing Contractor
ARDS 	Australian Regional Development Scholarships
AusAID	Australian Agency for International Development
DCP	Papua New Guinea-Australia Development Cooperation Program
DCS	2006-2010 Development Cooperation Strategy
DNPM	Department of National Planning and Monitoring
DPM	Department of Personnel Management
FAB	Facility Advisory Board
FCG	Facility Coordination Group
GoPNG	Government of Papua New Guinea
HR	Human resources
HRD	Human resources development
HRM	Human resources management
ICR	Independent Completion Report
IELTS	International English Language Testing System
JASC	Joint ADS Selection Committee
M&E	Monitoring and Evaluation
MC		Managing Contractor
MTDS	Medium Term Development Strategy
NTC	National Training Council
NTP	National Training Priorities
OECD DAC	Organisation for Economic Cooperation and Development -Development Assistance Committee 
PATTAF	Papua New Guinea-Australia Targeted Training Facility
PNG	Papua New Guinea
SSC	Scholarships Section Canberra
STT	Short Term Training
ToR	Terms of Reference
WDS		Workforce Development Strategy
Contents
Executive Summary	5
Introduction	7
Relevance (Rating: 4)	10
Effectiveness (Rating: 4)	12
Efficiency (Rating: 4)	16
Impact  (Rating: 4)	17
Sustainability (Rating: 4)	18
Gender Equality (Rating: 5)	18
Monitoring and Evaluation (Rating: 4)	20
Analysis and Learning (Rating: 3)	21
Evaluation Criteria Ratings	22
Conclusion and Recommendations	23
Annex 1. Terms of Reference	27
Annex 2. Persons Consulted	35

Executive Summary
Background and context
The supply of skilled personnel in PNG falls far short of demand. In particular, skills shortages are acute in the provinces following the devolution of management functions as a result of the Organic Law on Provincial and Local Level Governments. At the same time, ongoing budget constraints in PNG limit the local resources available to develop specialised training programs to meet new labour market needs. PATTAF was designed to support the broader governance and public sector reform objectives of the Papua New Guinea (PNG) - Australia Development Cooperation Program (DCP), by providing targeted training in areas that are high priority. PATTAF’s original contract pre-dated the Paris Declaration and Port Moresby Declaration on Aid Effectiveness, and the Accra Agenda for Action. 

Summary of PATTAF objectives, components and key results
PATTAF commenced in April 2002. The goal of PATTAF is to contribute to human resource development in areas of agreed focus in the PNG Development Cooperation Program. The purpose of the Facility is to provide targeted training in accord with the capacity and skills development needs of selected agencies. PATTAF has provided a broad range of services including management of AusAID’s tertiary study award programs (Scholarships) undertaken in Australia and the Pacific and workforce development services for selected GoPNG Public Service Departments. Due to a range of issues and circumstances, the contract for the delivery of the services provided by PATTAF has been extended a number of times and the requirements of this service have been substantially modified as AusAID’s scholarship program and workforce development requirements have evolved.
Overall, PATTAF appeared to have achieved what was asked of it. The lack of outcome level objectives in the original design impeded the ability to assess its contribution to broader development goals. However, the flexibility of PATTAF to deliver short term training was very useful for the wider PNG country program, in particular the public sector capacity building programs.  
The effectiveness of the scholarships component of the program was also not well defined beyond output-level, and post award support was neglected until recently. The performance of PNG scholars is of concern, as they have a high scholarship variation rate.
PATTAF performed well considering the types and levels of constraints. 

Brief outline of the evaluation findings 
Evaluation Criteria	Rating (1-6)	Explanation	
Relevance	4	While the original design of PATTAF lacked a focus on demonstrable outcome level objectives, the non-scholarship components have been very useful in responding to ad hoc tasking requests. The scholarship component lacked both contextually relevant objectives and delivery approaches that recognised local constraints.  	
Effectiveness	4	The lack of outcome level objectives in the design and consequently the M&E framework limited the definition of success and subsequent collection of information relevant to effectiveness.  However, in broadest terms, PATTAF’s has been regarded as highly ‘useful’ due to a flexibility to address many emerging needs that arose.  	
Efficiency	4	The facility delivered the majority of outputs required of it on schedule and within budgets. Quality of some scholarship outputs is lower than for other country programs (PNG has the highest number of scholarship variations), but constraints are arguably greater in the PNG context.	
Sustainability	4	While a lack of outcome-level monitoring of objectives (and outcome-level objectives themselves) hampered assessment of sustainability, generic assessments of impact were made by the program. These were ‘spot-checked’ by this ICR and were found to demonstrate ongoing utilisation of increased individual capabilities attributable to the facility. 	
Gender Equality	5	Of the total number of scholarships awarded, the percentage of female awardees is 50.5%. The performance of these females was high (less than half the failure rates of males), so this percentage does not appear to have been achieved by compromising quality of selection to date. However there is a concerning recent trend of a reduction of female applicants which will need to be considered in future efforts. 	
Monitoring & Evaluation	4	Outcome level objectives are virtually absent from the PATTAF design. However the program recognised the need for (and collected) some generic outcome level-information. 	
Analysis & Learning	3	Program staff suggested a number of worthwhile innovations over the course of the program, but a conservativeness within Government agencies usually prevented their adoption. Increased championing of such reforms by AusAID would have improved the situation.	
Rating scale: 6 = very high quality; 1 = very low quality. Below 4 is less than satisfactory.

Brief outline of lessons and recommendations:
(Specific recommendations avoided given existence of new design proposal)
Lesson 1: Regularly review and formally update designs as required during implementation.
Lesson 2: If a scholarship program is not getting the ‘right’ applicants in the first place, all subsequent aspects of the program will become more difficult to manage and ability to deliver desired outcomes will largely be impossible.
Lesson 3: In order to capture success at the outcome-level development programs (including scholarships) must be selective in their approaches.  
Lesson 4: In the PNG context, poor student performance is not likely to be able to be addressed solely though improved selection. 
Lesson 5: While flexibility is a highly valued aspect of the facility form of aid, if this flexibility is used in an ad-hoc manner, a loss of strategic direction and undesirable financial management burdens may result.
Introduction
Activity Background
General Information
The PNG-Australia Targeted Training Facility (PATTAF) was designed to provide management of Short Term Training (STT) and Australian Development Scholarships (ADS).  The purpose of the facility is “to contribute to human resource development in areas of agreed focus in the PNG Development Program”.
PATTAF commenced in April 2002 and will continue to operate in the current form until the proposed Scholarships PNG Facility design is implemented in March 2010.  The ICR covers the period from 2002 to August 2009.  GRM International was engaged by AusAID as the Managing Contractor for the PATTAF in 2002. 

Basic Activity Data
PATTAF has three main components:
•	Component 1 – Facility Management
•	Component 2 – Select and Place Suitably Qualified PNG Candidates in Long Term In-Australia Tertiary Courses
•	Component 3 – Identify and Deliver Relevant Short Term Training
Other components that were subsequently introduced to the facility:
•	Component 4 – Australian Regional Development Scholarships
•	Component 5 – Australian Leadership Awards Scholarships (ALAS)
•	Component 6 – Careers in Development Program
•	Component 7 – PNG-Australia Alumni Development Program
•	Component 8 – Strongim Gavman Program
•	Component 9 – Leadership Support Initiatives on HIV&AIDS

Over the life of the contract, Component 2 (Australian Development Scholarships) takes up 80% of the Facilities activity budget.  Basic information about results from the ADS component of PATTAF is summarised below:
Total Scholarships awarded from 2003 to 2008 (intake years)—773
Graduating awardees—479
Ongoing—210
Failure—84
Total male—385
Total female—388

Evaluation Objectives and Questions
The terms of reference for this evaluation are provided as Annex 1.
Evaluation Scope and Methods
General Approach Adopted
This ICR has been conducted under a specific set of circumstances. These included the fact that a major design mission had recently been completed in-country and a draft design subsequently formulated and progressed through AusAID’s peer review process.  While the function of an ICR is generally the same as most evaluative exercises, namely to address both accountability and continuous improvement (lessons learnt) concerns, the recent design mission had clearly already investigated the latter area to a far greater degree than is possible in an ICR. 
The implications of this for the ICR mission were that while the team needed to maintain the required focus on program performance, in attempting to draw lessons and forward-looking conclusions, care needed to be taken not to aggravate stakeholders by asking them the same things ‘all over again’.  In writing this report the team also had to avoid referring to any elements of the proposed new design as this could pre-empt eventual tendering processes.
These constraints were not regarded as unfortunate, as it was not appropriate for this ICR to attempt to create a competing design model.  Hence, in terms of extracting lessons learnt this ICR limited itself to identifying a list of high-level program functions or aspects that stakeholders thought were critical to either retain or modify.  This list may therefore be used as a simple ‘double-check’ on the proposed design direction, rather than a redundant or competing set of detailed suggestions.  Specific recommendations have been avoided to avoid the creation of competing design models.

Specific Methodology
Given the range of considerations to be addressed by this ICR and the fact that many of them are quite complex constructs unsuitable for direct reference in respondent questioning, the SWOT (Strength, Weaknesses, Opportunity and Threat) framework has been applied as a simple, but robust means of structuring information gathering. The SWOT framework ensures that any issues or observations raised by respondents are tested for relevance to program progress or performance.  SWOT is also an effective means of identifying the key issues as perceived by respondents compared to necessarily more prescriptive and comprehensive design-focused questioning.
In recognition that a significant percentage of the information available to the team would be derived from interviews or focus group discussions (and therefore largely qualitative), the SWOT approach was combined with the basic structure of a ‘Qualitative Compilation Table’. This compilation table approach is based on the collection of pertinent structural data with a consistent set of related dependent information.  In practice, this means firstly allowing respondents to raise any SWOT issues (the ‘structural’ data) that “they think are important”; either in an (initial) unprompted part of an interview or in relation to suggested discussion topics.  Once an issue is raised, the collection of ‘dependent’ information means asking a consistent set of questions about that issue.
For the purposes of this ICR the prompt topics applied were:
•	Poor student performance
•	Male vs. female performance
•	Taskings of PATTAF
•	Government awardee conditions
•	M&E issues
•	Design (existing) issues
•	Resourcing/management issues
•	Selection processes
•	Setting of training/general priorities
•	Decision making (program governance)

The set of consistent follow-up questions applied to each SWOT issue raised were:
a.	How did this issue affect the PATTAF Program?
b.	Who were the main players in relation to the issue?
c.	What is the suggested response to the issue? (or if it has already been resolved, how was it resolved?)
The information compiled in this way was then combined with information derived from document review and addressed under relevant ICR format headings.  Findings were also synthesized into a coherent set of lessons learnt.
Interviews were conducted with AusAID staff, GoPNG Representatives, alumni focus groups, PATTAF staff, Alumni Association representatives and employers of alumni (a list of persons consulted is provided as Annex 2).
In response to a perceived shortcoming of current M&E systems (see M&E section), an additional triangulation exercise was conducted when meeting with employer representatives.  This simply consisted of requesting specific examples policies of practices that alumni were now working on, which they would not have been tasked with had they not undertaken the study facilitated by a scholarship. The sample size obtained in this manner was tiny, but this limitation is not inconsistent with the ‘spot check’ function of an ICR.

Evaluation Team
The evaluation team consisted of three members.  Two (including the team leader) were independent consultants. One of these was an international consultant, chosen for past experience in AusAID scholarships reviews and designs. The other was a national consultant chosen for experience in the PNG development context. The third team member was an Program Coordination Officer from the PNG Branch of AusAID with significant Melanesian field experience. 
The inclusion of an AusAID officer on an ICR team has the potential to generate perceived conflicts of interest, but these were addressed early in team discussions and appropriate roles assigned to relevant members to avoid realisation of such conflicts. 

Evaluation Findings
Relevance (Rating: 4)
Design Issues
The longevity of PATTAF has meant that the original design was a ‘first generation’ approach to a facility.  This early approach to facilities was characteristically process-based and lacked a focus on demonstrable outcome level objectives (as more recently prompted by agreements such as the Paris Declaration and Accra Statement).  The PATTAF design was no exception to this tendency.  Consequently, the ‘objectives’ of its original logframe were stated at output level, and would not be considered objectives under current AusAID definitions.  
Without this appropriate statement of the specific desired outcomes, it is hard to judge whether the PATTAF design was relevant to the specific needs and constraints of the delivery context.  Hence, the assessment of PATTAF design relevance is only possible in a more general sense.  The program’s higher-level purpose statement proposes addressing capacity development. It is undeniable that there was a need for this in the PNG context.  However, as many donors are now beginning to realise, capacity development should be regarded only as a means to an end, rather than an end in itself.  That is, the desired effects of such capacity development need to be stated if aid effectiveness is to be correctly addressed.  In the case of PATTAF, some attempts to contextualize capacity development was made by reference to the Workforce Development Strategy (and later on, the Medium-Term Development Strategy and similar documents), but this still lacked any real indication of the specific effects sought.  The design therefore largely failed to answer the key relevance question of ‘why are we doing this?’

In-Practice Experience 
While the design’s approach to capturing relevance therefore appears flawed, the in-practice experience of stakeholders tells a different story.  The overwhelming feedback from all stakeholders has been that the program has been very useful in the PNG context.  However, most of this praise is directed at the non-scholarship components of the program tasked in the last three years of the program, and this usefulness was primarily regarded as having resulted from the flexibility afforded by the facility approach, especially that provided under Service Order and other flexible tasking arrangements.  In contrast, the in-practice relevance of scholarship components was commonly regarded as being hamstrung by a one-size-fits-all approach to delivery (discussed in detail under the Effectiveness Section).  
The tasked components of the program were able to regularly ‘borrow’ relevance from the initiatives (AusAID and/or GoPNG) that they were ‘sponsored’ by.  Hence, to some extent, the lack of outcome-level consideration in the facility design seems to have been compensated for by later relevance consideration that went into designing these sponsoring initiatives.  It is therefore not correct to say that the program as a whole lacked relevance, and providing a rating of less than 4 would inappropriately hide the unusual level of enthusiasm and appreciation afforded the program across all stakeholders.  
Clearly, the program got some things right.   Some credit for this must also go to the staff of PATTAF, particularly in relation to their sound knowledge and understanding of the PNG context.  While the diversity of tasking requests PATTAF dealt with meant that there were cases in which these staff were operating outside their core areas of expertise, in most instances the contextually informed nature of implementive decisions made by the facility aided overall relevance. 
In regard to the overall relevance of the facility model used, feedback from AusAID posted and local staff would strongly support that the flexibility offered by this model was very useful.  While it would not be necessary if the program was solely focussed on delivering scholarships, if a broader and more flexible focus is required by the program, a facility model remains appropriate.

Roles and Responsibilities
Roles and responsibilities related to delivery of scholarship components were reasonably well defined and executed, although exceptions were encountered in the management of activities associated with the Alumni Association (see Effectiveness Section) and GoPNG ability to meet the commitments imposed by the Facility Advisory Board, Facility Coordinating Group and Joint ADS Selection Committee (JASC). In relation to the JASC, some feedback indicated that this may have been exacerbated by somewhat unrealistic expectations of agencies being able to commit staff for two separate full fortnight blocks. The appropriateness and efficiency of selection committees undertaking eligibility checks was also questioned. 
The roles and responsibilities associated with tasked activities was less well defined and caused significant problems in practice, especially in relation to imposing excessive financial management burdens on the AusAID initiative manager, and a lack of shared or consistent approaches to M&E.  Due to the broad range of tasks given to the facility, this lack of clarity in roles and responsibilities also extended to relationships between managing contractors (i.e. PATTAF and the MCs of programs that sponsored tasks).  Such relationships sometimes tended to be competitive rather than cooperative, and were therefore significantly less productive than they might have been.  

Scholarship Components
To put the overall rating of 4 for relevance in proper perspective, it needs to be stated that if this ICR were exclusively considering the scholarship components of the facility, this score could easily fall 1 to 2 points.  As with many past scholarships programs (both within AusAID and other donors), a ‘one-size-fits-all’ approach was adopted to delivery of the ADS component of PATTAF.  While such standardised approaches may be administratively convenient, they inherently impede defining of specific contextually relevant objectives, and the adoption of delivery approaches tailored to local constraints.  Unfortunately, contextually relevant objectives and delivery approaches that recognise local constraints are probably the two most important precursors to the success of any development initiative (especially in an aid effectiveness sense). 
The sources of this over-standardisation are numerous.  Some, such as rules that students may only choose undergraduate courses not offered in PNG (regardless of significant differences in quality or specific content of courses) are derived from AusAID’s Scholarship Section Canberra (SSC) guidance. Another source appears to be a generalised conservatism within GoPNG agencies preventing consideration of innovative approaches to problems, even if such approaches would likely be beneficial to all parties. Related to this, and probably equally culpable for any shortcomings, was an apparent reluctance of the AusAID Post to strongly champion such reforms with GoPNG.  
An example of such proposed reforms was a plan for PATTAF to canvass individual government (and private) agencies to determine and respond to their specific priorities under the MTDS (and the various higher-level skills assessments available over the course of the program).  Responding to these priorities would have simply involved giving selection bias to applicants whose choice of course coincides closely with the specific priorities of their employers.  Note that this is a perfectly legitimate use of awardee selection criteria and one that will always promote significantly greater relevance.   Facility staff put forward this example and a number of other suggested reforms over the life of PATTAF, but for reasons already provided above, these were not progressed further.  This is not to say that the ICR team agreed with every reform proposed, but to give credit to PATTAF staff, many of these suggested reforms mirrored those recommended by recent SSC reviews into delivery of AusAID scholarships.  It must also be said that feedback from stakeholders included examples of facility staff going ‘above and beyond the rules’ to accommodate specific needs of agencies or individuals and examples where rules where applied in an excessive manner when it suited an administrative purpose.
Effectiveness (Rating: 4)
The ToRs for this ICR required a focus on the scholarship component of the program.  Other components were still considered in assigning the above effectiveness rating, but this consideration was limited to their higher-level effects on the program, rather than the detailed effectiveness of each non-scholarship task.
Measuring Effectiveness
The output-level achievements of the program have been reported elsewhere in this report, as have general findings regarding general impact (see Figure 2). The lack of outcome-level objectives in the design and subsequent inability of the M&E framework to define related indicators means that outcome-level of information is unavailable.
Despite the lack of specific higher-level measures, there was substantial positive feedback from AusAID and GoPNG respondents about the facility’s effectiveness. This feedback was usually focused on the non-scholarship components of the program, and was also generally related by respondents to the level of flexibility the facility model provided.  The ability to address emerging needs of other AusAID or GoPNG initiatives in a timely manner was seen as having boosted the effectiveness of those sponsoring initiatives.  Unfortunately, the lack of a shared M&E platform between the sponsoring initiatives and the tasks carried out for them by the facility makes it very difficult to substantiate whether real outcome-level effectiveness was achieved or whether the perception of effectiveness was limited to the program’s ability to deliver outputs.  Given that the latter perception is well collaborated by program records and that the program logframe is limited to output-level objectives (including for scholarship components), it is also difficult to argue that the facility failed to deliver what was asked of it in the majority of cases. Procurement processes and provision of technical assistance in relation to tasked activities were regarded as timely and of generally high quality, albeit somewhat expensive.
What makes it difficult to award more than the rating provided above (4), is that current standards of assessing effectiveness inherently require some indication of outcome-level effectiveness.  The impact of original design flaws in facilitating or measuring outcome-level effectiveness is therefore also reflected in this rating.

Scholarship Components
Were the above rating to only consider scholarship components of the program, it would need to be reduced by at least one point (i.e. to 3 or 2).  That the program delivered against the process-related outputs required for this component by the logframe is well substantiated by program records. Hence, it is difficult to argue that the managing contractor did not do what was asked of it.  The need for a reduction in a scholarships-specific rating is again largely design-related.  Not only was effectiveness of scholarships never defined by the design beyond output-level, post-award measures to support returning students were almost entirely lacking.  While very common in older scholarship programs, this process-driven approach to scholarship design and delivery has been identified as a major impediment to demonstrable effectiveness by a number of recent SSC global reviews.  The inability of AusAID scholarship programs to demonstrate such effectiveness was also noted in the 2008 OECD DAC Peer Review of the Agency.  Given the key processes of course delivery end once an awardee completes their course and returns to their home country, these types of flawed design tend to neglect post-award support to students. This is unfortunate from an effectiveness perspective, as it is usually only post-award that scholarship outcomes are delivered.  In short, from a return for investment perspective, this amounts to “dropping the ball once you reach the try line”.  
To be fair to both program and AusAID staff, it must be acknowledged that most were fully aware of these design problems and did try to go beyond merely what was required. On the PATTAF side, this included development of a Performance Assessment Framework and conducting of Most Significant Change workshops with returnees, both of which attempted to collect some information about the effects of scholarship provision. The extent to which the program could extend into these additional activities was constrained however, as activities not covered by the design or formal taskings were necessarily under-funded. 
It should be noted that support provided to the Alumni Association to date has primarily been in the form of ‘seed’ or ‘core’ funding to aid establishment of the organisation. While some notable achievements have resulted from Alumni Association efforts and many other ideas have been mooted, no systematised proposals for constructive functions of the Alumni Association have yet been self-generated.  
Conflicts have occurred between PATTAF and the Alumni Association.  These are easily understood in the context of competition of between the Association’s wish for autonomy (including financial autonomy) and the facility’s (and AusAID’s) wish to maintain development relevance of alumni activities and meet AusAID financial accountability requirements.  Neither of these players is without some fault in this conflict. The managing contractor is reluctant to give up control of program resources for a variety of reasons. The Alumni Association has been slow to develop functions which clearly deliver development effectiveness benefits (or concrete proposals for such functions), and has largely been interested in attempts at growing the organisation for it’s own sake.  This situation suggests the need for direction to be provided by AusAID through any future design, clarifying the roles, responsibilities and obligations of both parties.  In short, it may be possible to provide greater financial and other autonomy to the Association, but only if it comes up with worthy proposals to fund.  If Post resources allow, it would clearly create a better dynamic between the players if Association generated proposals are submitted directly to AusAID for approval, and the Managing Contractor simply provided financial oversight (rather than both financial and directional oversight).  This would give a greater sense of autonomy, but also reinforce the need to proactively develop relevant functions of the Association to access funding.  There is a real need for the Association to adapt it’s approach to growth, such that both membership and development usefulness of the organisation expands concurrently. This is the only way to improve sustainability, whether in terms on ongoing access to external funding or developing self-sufficiency. 
The measures of effectiveness used in the Performance Assessment Framework were primarily respondent ratings against twenty ‘Critical Success Factors’.  On the whole, the ratings provided in this way tended to support the hypothesis that the scholarships provided through the facility were having a positive effect on recipients and that these were able to be translated into positive effects on workforce development in PNG (see figure 2).  Given the subjective nature of this information, the ICR team attempted to triangulate this finding by requesting employers to give concrete examples of policies or practices that alumni were now undertaking that they would not have been tasked with if they had not undertaken the studies facilitated by a scholarship.  While the sample size for this exercise was tiny, relevant examples were forthcoming in all cases.  This finding therefore allows more confidence in the validity of the findings of the Performance Assessment Framework.  However, tying such general findings back to positive effects relevant to the MTDS (or earlier Workforce Development Strategy) is problematic.  Had the design been more specific in the effects it wished to generate, this would have been easily achieved using the above types of approaches, but the vagueness of the current design in regard to outcomes currently precludes it.

Performance of Scholars
Given the logframe’s focus at output level, the comparatively high failure (or reduced achievement) rate of PNG awardees cannot be ignored in a discussion of PATTAF effectiveness.  In most recent years the key determinants of poor performance have been maleness and age (younger individuals perform less well).  There is also a less obvious trend of poorer performance from the public service scholars, but the true magnitude of this trend is difficult to determine, given leakage between ‘public’ and ‘open’ categories in their early usage. While some of these determinants may be unsurprising, as an inter-related set they bring the failure/reduced success rates for PNG scholars to the highest of any AusAID program in the world.  The ICR team specifically investigated reasons for this relatively poor performance, but no single factor was dominant in responses.  To state this another way, for a wide range of cultural, personal and institutional reasons, a significant proportion of males, the lower age groups and,  quite likely, public servants appear to poorly equipped to undertake scholarships than equivalent candidates from most other countries.  This finding suggests the need for a more tailored approach to provision of AusAID scholarships in PNG.  
There would appear to be two possible responses to this problem.  The first option would be to alter selection to avoid sending these groups. The second would be to provide more oversight and support to selected PNG students (especially while on award) to better equip them to succeed.  The first option may be feasible for some factors, but large- scale avoidance of male or younger awardees is unlikely to be feasible or desirable in the long term. The second option has obvious resourcing implications, but these may prove a worthwhile investment if the greater oversight can reduce both management effort and costs associated with poor performance.  Given that the most common implication of poor performance are costly extensions, obtaining a clear return on the investment may be quite achievable, so this option clearly deserves further consideration in any future design.  
It is interesting to note that alumni focus groups were particularly keen to see such oversight not just take the form of higher levels of assistance to those who need it, but also include lower levels of tolerance of poor performance (or behaviour) and stricter enforcement of AusAID policies or program charter agreements.  This ‘lower tolerance’ perspective appears to be borne of a realisation that scholars who are poorly performing due to a lack of personal commitment are effectively wasting a chance to study that might have otherwise have been taken advantage of by another, more deserving, PNG national. There was also the strong view that such individuals are giving other PNG students (and the Nation as a whole), a bad name; with direct effects on future students in relation to numerous matters (e.g. obtaining accommodation, finding employment, etc.).  Government stakeholders were also very supportive of both greater support and lower tolerance approaches.  As many government agencies continue paying salaries to employees while they are on scholarships, the cost implications of extensions are as serious for these agencies as they are for the program.  

Program Governance
Program governance appears to have been adequate, although the operation of two high level bodies (viz. the Facility Coordinating Group and the Facility Advisory Board) appears to have been too resource intensive for GoPNG to fully support.  Limited attendance or lack of continuity in delegates led to these two boards eventually being combined in practice, with little change in overall results. 

Scholarship Delivery Options
Given the questions related to current quality of many PNG tertiary institutions and the need to ensure that sponsored students do not simply displace less-privileged local students, the option of providing scholarships within PNG requires more consideration than the scope of this ICR allows.  While local provision (outside the ARDS framework) deserves careful consideration, other local delivery options also exist which overcome both issues mentioned above.  These options are primarily based upon the delivery of courses offered by Australian institutions in PNG.  This may involve arrangements between Australian and PNG teaching institutions or a simple ‘importation’ of the course, including Australian teaching staff and associated resources.  Bringing the course to the students has obvious potential for cost savings and removes many of the barriers otherwise faced by potential applicants.  Both long and short courses can be accommodated by this approach and it has the benefit of ensuring Australian standards are maintained in provision of AusAID Awards.  
Offering of short-term courses in this way (or in Australia) are a particularly attractive option from an effectiveness perspective, as feedback from both employers and potential awardees in both the private and public sectors in other recipient countries has stressed the fact that the key players in these sectors are simply unable to leave their posts for long periods of time without becoming redundant.  In recognition of this fact, AusAID is currently attempting to better systematize its ability to offer Australian standard short course awards.
Existing program efforts in relation to mixed mode delivery of courses strongly supports the view that these approaches can have significant benefits to both awardees and the program. Such options need to be considered by any new design.
Efficiency (Rating: 4)
The assessment of the Efficiency of the program for the ICR was limited to the selection of focus group discussion, a sample of employers from both the public and private sectors and reports provided by both AusAID and PATTAF management.  The draft Rating of 4 is based on these findings.
The original design of the program under Phase A had three components.  The efficiency assessment looks primarily at two of these three components. These are Component 2: “Select and Place Suitably Qualified PNG Candidates in Long Term In-Australia Tertiary Courses” (for Australian Development Scholarships) and Component 3: “Identify and Deliver Relevant Short Term Training”.  As the program evolved, the facility was given a new responsibilities and a new component was created. This posed some management issues for both AusAID and the contractor.
Australian Development Scholarships
According to the most recent PATTAF report (August 2009), of the 903 awardees selected from 2003 to 2009, 479 completed courses (439 completed their original courses).
In recent SSC findings, PNG had the highest number of variations compared to other country programs.  The cost to the program (and AusAID) of the variations from 2003 to 2009 is $7.1 million.  As illustrated in Figure 1, of the total variation cost 81% (approximately $6.5 million) was for course extensions, main reason being failure in one or more academic subjects.  
Figure 1:  Variation Costs -  2003 to 2009
(Note: This information replaces a chart that is available in the PDF version of this report)

Extension—81 per cent
Reduction—Five per cent
Sup support—Two per cent
Termination—Seven per cent
Transfer—One per cent
Other (family)—Three per cent
Course detail change—One per cent
Suspension, Withdrawal and Deferral—Zero per cent

It was expressed by various stakeholders (including former awardees), that more stringent measures should be taken to discourage extensions.
There has been a change in the ADS Policy to limit extensions to one semester only.  There is already evidence of this working.  In 2009, there were 22 extensions granted, the lowest for the program (2008: 33).  The highest was 61 in 2007.
In terms of value-for-money considerations, the program has consistently delivered what was requested of it largely within component or task budgets.  There was anecdotal feedback from AusAID staff that estimated costs associated for some tasked items were relatively high, but not sufficiently so to discourage use of the facility.  The level of staffing appears consistent and comparable with similar AusAID scholarships programs of the same scale, and the fact that a number of these same staff also expended considerable effort in managing the tasked components of the program suggests that they used their time and other resources efficiently.
Impact  (Rating: 4) 
The long-term impact of the Scholarship component of the program in terms of sustainable outcomes is difficult to assess. Former awardees stated that they had increased capability upon return.  The most common comment (especially from females) was the significant increase in their self-esteem and confidence which directly contributed to improved performance in the workplace.  It would be fair to say that the impact was therefore more significant at a personal level for most awardees.  This impact extended to family members for those whose families accompanied them for the duration of the course. 
From an impact study conducted by PATTAF in 2007 (see Figure 2), 40% of the awardees that took part in the study responded that they returned to their position.  On the other end of the spectrum, 12% were unemployed and 3% were demoted in their organisations.
The comments received from private sector employers indicated that graduates with qualifications and work experience abroad were preferred over local graduates as 

overseas graduates have international experience and a mind set that is generally gives them an advantage in the work place.
No specific impact comments were received from the Public Sector agencies that were consulted in relation to the Scholarship Component or other training initiative offered through PATTAF.  However, there is evidence of graduating awardees holding key positions in various government departments and agencies.  It is widely acknowledged that the public sector lacks a well trained and efficient workforce.
One of the impacts of the program is that as individuals become better skilled and qualified; they are marketable within the country as well as overseas.  This poses a risk for both the private and public sector that already have a limited pool of quality people to choose from.  The public sector especially is losing its good people to the private sector because of more attractive work conditions offered by the private sector.
Figure 2 – ADS Development Impact
(Note: This information replaces a chart that is available in the PDF version of this report)
Returned to my position—40 per cent
Changed employer and position—14 per cent
Promoted in my organisation—28 per cent
Demoted in my organisation—6 per cent
Unemployed—12 per cent

Sustainability (Rating: 4)
In order for the short term outcomes to be sustainable for the long term, there needs to be a more co-ordinated approach and strategic approach by the key stakeholders, mainly the Public and Private Sectors, the Program Management (AMC or ISP) and AusAID.  PATTAF management have attempted to address through Tri-partite Agreements (or ‘Charters’), which require the agency (in the public sector) or the company (in the private sector) to make an employment commitment to the awardee, with the awardee reciprocating this commitment to their employer (and the program).  
In both the public and private sectors (but especially in public), there is lack of appropriate HRM and HRD plans. If they do exist, they do not provide specific direction useful at individual level.  During the review, the team had this concern expressed by two key agencies, the DNPM and the NTC.
Gender Equality (Rating: 5)
PATTAF has met gender targets far more effectively than similar AusAID programs in other countries.  Overall, 50.5% of scholarship recipients were female. This statistic alone does not directly reveal whether there was effective targeting of appropriate women, or whether there was merely a default willingness to select those that did apply. However, the performance of females was high (less than half the failure rates of males). This suggests meeting the gender targets does not appear to have compromised the quality of selection of female students to date.

A worrying output-level statistic which has arisen in latest intakes is a reduction in the number of applications for scholarships received from females.  This trend is of concern from both a gender equity perspective and in relation to future overall student performance outlooks.  While some changes in application patterns may be related to restructures taking place in government agencies, the consensus of the female alumni focus group consulted by the ICR team was that this reduction in applications from females is a direct reaction to the removal of family-related stipends and other support from AusAID scholarship programs.  While there has been a lag between the removal of such benefits and the reductions in female applications, the group suggested that this is easily explained in that ‘word has now had a chance to have gotten about’.  If this trend continues, it is unavoidable that, in order to maintain gender targets, lower quality female candidates will need to be provided awards.  This will again flow through to exacerbated student performance problems.
While the above example is primarily a gender equity issue, a similar shortfall in applications from provincial areas has long been acknowledged by program reporting.  A key finding of recent SSC reviews is that while a ‘one-size-fits-all’ approach to provision of scholarships seems administratively convenient, it oversimplifies approaches to a point that they are no longer effective. Any oversimplified approach is based on false assumptions and therefore highly prone to unintended consequences, problems or failure.  The most common oversimplification made in the context of scholarships is that it is enough to simply ‘offer them’.   Regardless of how ‘freely and openly’ scholarships are offered, there are always a raft of unintended barriers to application either in already in existence due to the delivery context or created by the scholarship program’s own administrative systems.  
In the PNG case, the reduction of applications from women appears to be the direct result of a barrier imposed by an administrative decision made by SSC to remove AusAID assistance for family accompaniment.  Focus group discussions clearly suggest that a large proportion of PNG women simply will not accept (or even apply for) a scholarship if they are aware that they are not enabled to at least take their children along.  While the removal of this form of assistance does not mean that women are ‘not allowed’ to take their children with them, enough women have now attempted to do so in the absence of the additional assistance to get the message back to others that it is very difficult.  For this issue to be addressed, the Post will need to raise this issue with SSC and the AusAID Gender Policy Section to communicate the implications of retaining the policy in the PNG context.  
In the more general context, the barriers that are relevant to a particular individual will change with factors such as sex, location (urban/rural), culture, religion, ethnicity and many others.   In many of the cases examined by the global reviews, the individuals most severely affected by such barriers were those from whom applications were most desired (viz. women and persons from non-urban areas).  
In any context, obtaining the desired awardees means obtaining the right applicants in the first place.  Obtaining appropriate applicants will clearly have highly beneficial effects for all subsequent management and impact aspects of a program. Obtaining the right applicants will always, to some extent, mean ‘enabling’ those whom you want to apply to do so.  Whether in the context of women or people from outside the National Capital District (and the other main urban centres) ‘enablement’ can include things as simple as more targeted promotion (to ensure the right people are reached), or measures as administratively heavy as tailored allowances.  There are resource implications of putting in place such enabling measures and the choice of those that can and cannot be implemented in a particular context must be made in recognition of the resource constraints of the program.  However, given that obtaining the ‘right’ candidates in the first place has very significant beneficial flow-on effects for every subsequent part of the scholarship process (including post-award effectiveness), the return on some of these investments may well outweigh their cost.  In other words, putting a little more effort into some of the ‘up-front’ processes may save a lot of work later on. Such options should be considered by any future design.
While it is inappropriate for this ICR to address potential design detail, the SSC reviews and associated guidance referred to do contain more specific direction in this regard and might be referred to in finalising and implementing any new PNG Scholarships design.   
Monitoring and Evaluation (Rating: 4)
This is clearly a ‘first generation’ facility design.  The objectives included in its logframe are clearly focused on delivery of outputs; many of which are then used as inputs to other AusAID programs (especially after the transition to use of tasking by Service Orders). Outcome-level objectives are virtually absent from the design.  This is not surprising, given that this design’s formulation preceded the Paris Declaration, Accra Statement and the many other recent initiatives that focus on ODA effectiveness.  However, it does mean that in today’s terms the program’s design is very dated and future designs could not repeat this approach.  Any new version of this program will face the necessity of defining and demonstrating delivery of (modest) developmental effects.  This need has been reinforced by the recent OECD DAC Peer Review findings, which single out AusAID’s scholarship program for specific comment in regard to a need to better assess impact.
The M&E frameworks applied to this program are a sensible and effective response to the logframe requirements.  However, as the objectives of a program are the main starting point of any M&E framework, the issues associated with the original design of the logframe have also clearly impacted upon overall the scope of the M&E applied. The lack of any clear direction in regard to expected outcome-level achievements of the program meant that the implementers could not easily identify relevant outcome indicators, nor allocate significant resources to their collection.
In other words, the lack of outcome-level objectives in the logframe has meant that ‘what constitutes success’ beyond delivery of outputs was never well defined.  While the program commendably recognised the need for (and collected) some outcome level-information, the lack of clear design direction as to what these should be necessitated resorting to use of largely abstract measures as indicators. An outcome-level element of the overall M&E framework was eventually introduced (viz. the Performance Assessment Framework, 2007), but relied almost exclusively on subjective ratings by alumni or their employers against selected criteria. While such ‘subjective quantification’ may have a place in higher-level synthesis of information and overall program reporting, its use as a primary source of base data partially defeats the main purpose of M&E; namely, to objectively inform independent readers and allow them to form their own opinions. The alumni opinions collected by the Performance Assessment Framework remain useful, but lack a truly defensible ‘evidence base’, and would have been better used in conjunction with some form of modest, but verifiable outcome measures.
The program has also very recently conducted a number of Most Significant Changes workshops with alumni.  This approach is again useful for collecting general information about effects, but used alone would not be sufficiently targeted to consistently inform indicators of relevance to specific desired effects associated with the MTDS (had these been defined).  
The ICR team is not intending to sound overly critical of the existing PATTAF M&E framework, as improvement of the M&E framework of this program would have first required revision of the existing outputs-focused design. Updated M&E guidance for this type of program that specifically addresses the above issues is now available through SSC.  This guidance offers pragmatic suggestions in relation to both defining realistic outcome-level objectives and identifying modest outcome-level indicators with which to monitor them.  
Analysis and Learning (Rating: 3)
As discussed under the relevance heading, the effects of over-standardisation of approaches and other issues clearly indicated the need for reforms to the program. While program staff did identify and suggest the need for appropriate reforms, these were rarely implemented. A key constraint appears to have been a generalised conservatism within GoPNG agencies preventing consideration of innovative approaches to problems, even if such approaches would likely be beneficial to all parties. Related to this, and probably equally culpable for any shortcomings, was an apparent reluctance of the AusAID Post to strongly champion such reforms with GoPNG.  
An example of such proposed reforms was a plan for PATTAF to canvass individual government (and private) agencies to determine and respond to their specific priorities under the MTDS (and the various higher-level skills assessments available over the course of the program).  Responding to these priorities would have simply involved giving selection bias to applicants whose choice of course coincides closely with the specific priorities of their employers.  Note that this is a perfectly legitimate use of awardee selection criteria and one that will always promote significantly greater relevance.   Facility staff put forward this example and a number of other suggested reforms over the life of PATTAF, but for reasons already provided above, these were not progressed further.  This is not to say that the ICR team agreed with every reform proposed, but to give credit to PATTAF staff, many of these suggested reforms mirrored those recommended by recent SSC reviews into delivery of AusAID scholarships.  It must also be said that feedback from stakeholders included examples of facility staff going ‘above and beyond the rules’ to accommodate specific needs of agencies or individuals and examples where rules where applied in an excessive manner when it suited an administrative purpose.

Evaluation Criteria Ratings
Evaluation Criteria	Rating (1-6)	Explanation	
Relevance	4	While the original design of PATTAF lacked a focus on demonstrable outcome level objectives, the non-scholarship components have been very useful in responding to ad hoc tasking requests. The scholarship component lacked contextually relevant objectives and delivery approaches that recognised local constraints.  	
Effectiveness	4	The lack of outcome level objectives in the design and consequently the M&E framework limited the definition of success and subsequent collection of information relevant to effectiveness.  However, in broadest terms, PATTAF’s has been regarded as highly ‘useful’ due to a flexibility to address many emerging needs that was arose.  	
Efficiency	4	The facility delivered the majority of outputs required of it on schedule and within budgets. Quality of some scholarship outputs is lower than for other country programs (PNG has the highest number of scholarship variations), but constraints are arguably greater in the PNG context.	
Sustainability	4	While a lack of outcome-level monitoring of objectives (and outcome-level objectives themselves) hampered assessment of sustainability, generic assessments of impact were made by the program. These were ‘spot-checked’ by this ICR and were found to demonstrate ongoing utilisation of increased individual capabilities attributable to the facility. 	
Gender Equality	5	Of the total number of scholarships awarded, the percentage of female awardees is 50.5%. The performance of these females was high (less than half the failure rates of males), so this percentage does not appear to have been achieved by compromising quality of selection to date. However there is a concerning recent trend of a reduction of female applicants which will need to be considered in future efforts. 	
Monitoring & Evaluation	4	Outcome level objectives are virtually absent from the PATTAF design. However the program recognised the need for (and collected) some generic outcome level-information. 	
Analysis & Learning	3	Program staff suggested a number of worthwhile innovations over the course of the program, but a conservativeness within Government agencies usually prevented their adoption. Increased championing of such reforms by AusAID would have improved the situation.	
Rating scale:
Satisfactory	Less that satisfactory	
6	Very high quality	3	Less than adequate quality	
5	Good quality	2	Poor quality	
4	Adequate quality	1	Very poor quality	

Conclusion and Recommendations

Lesson 1: Regularly review and formally update designs as required during implementation
While not a good-practice example of a lesson (in that it amounts to a ‘motherhood statement’), this is probably the most obvious lesson learnt by the PATTAF experience.  The ‘design issues’ that affected the program were really a reflection of the changing aid priorities, and a reminder that it is not just changes in local context that need to be allowed for when building ‘responsiveness’ into a design.  Annual reviews that were built into PATTAF implementation either did not provide sufficient opportunity for change or were not used to their full potential.
While it is acknowledged that imposing significant design amendments on ongoing programs can have very significant administrative, financial and contractual implications, not doing so when necessary defeats development effectiveness and therefore amounts to ‘allowing the tail to wag the dog”

Lesson 2:  If a scholarship program is not getting the ‘right’ applicants in the first place, all subsequent aspects of the program will become more difficult to manage and ability to deliver desired outcomes will largely be impossible. 
The implication of this lesson is merely that it is always worth putting some up-front effort into obtaining the right applicants.  This may simply mean having the program explicitly define a range of ‘desired applicant profiles’, so that promotional strategies can ensure that these people are reached.  It might also mean removing barriers that may be reducing the numbers of applications received by certain target groups (e.g. females and provincial dwellers).
Note that defining desired applicants does not immediately imply reducing the pool of available candidates.  Rather, the targeted promotion it facilitates can translate into replacing ‘unwanted’ candidates in the existing pool with a higher proportion of desirable candidates. 

Lesson 3: In order to capture success at the outcome-level development programs (including scholarships) must be selective in their approaches.
This is again stated in a ‘motherhood’ sense, because it relates to a number of factors relevant to PATTAF.  Firstly, sufficiently specific outcome-level objectives are a necessary precursor to achieving development effectiveness.  Fail to define success in this way, or define it in too general terms, and identifying attributable indicators to demonstrate achievement is impossible. Secondly, provided they align with objectives, showing a bias in promotion, support provided or selection processes is both necessary and appropriate in development-oriented scholarship programs.  Thirdly, mechanisms that allow scholarship programs to be selective early in candidate acquisition are usually worth the investment.   
The first two factors have been discussed in detail elsewhere in this report, but it is worth recalling that they again have implications for areas such as gender equity and other inclusiveness criteria.  The third factor has also already received some attention in other sections, but it is worth adding here that on the most basic level this may mean adding mechanisms that allow selection to be more effective.  The obvious example in the PNG case is the need to do some form of face-to-face vetting of short-listed candidates.  As confirmed by the shared experience of all AusAID scholarship programs, written applications alone are not a sufficient means for separating candidates.  Not only are they easily ‘fudged’, many of the characteristics that might determine the suitability of a candidate (e.g. attitudes, personality, etc.) are not something that can be gleaned from a written document alone. Face-to-face encounters do not necessarily imply standard single candidate interviews.  PATTAF has recently introduced the use of group interviews for Careers in Development (CiD) candidates. Such options may not only be less resource intensive, but may also be more effective in revealing the key characteristics of participants. Consideration needs to be given to adopting this approach more widely in relation to PNG scholarship programs.

Lesson 4:  In the PNG context, poor student performance is not likely to be able to be addressed solely through improved selection
It is possible to identify most poorly performing groups within cohorts as males, public servants and younger candidates.  While it may be possible and warranted to shift some scholarship numbers between the public and private sector, reallocation on the basis of sex and age would skew outcomes in a manner which may not be sustainable in the long term.
Given discussions in preceding sections, provision of greater on-award oversight to PNG students appears warranted both in relation to improving study outcomes and on a cost-benefit basis by reducing expensive extensions.  Any such oversight needs to include both additional assistance and increased enforcement of general AusAID rules or specific program charter agreements. 
While an in-Australia presence is likely to be required to address some oversight requirements, an issue raised to the program team by a number of respondents may also suggest another relevant reform.  This issue was that of students not providing their results to either their employers (especially government agencies) or the program as required or in a timely manner.  One source of this problem seems to have been the blind application of privacy policies by Australian teaching institutions.  While focus groups of alumni indicated they thought that forwarding results was the responsibility of the institution, the institutions in turn were refusing to do so on privacy grounds.  While these institutions are right not to make student results available to ‘just anyone’, the scholarship agreements and charters that these student sign amount to a conferring of permission to the program to access this information.  Government agencies sending scholars also require similar agreements.  Even at the SSC level, AusAID has had difficulties explaining this to Australian institutions, and a number of respondents to the ICR pointed out that there is a perverse incentive for such institutions not to report student’s problems early, because they significantly benefit from the increased revenue generated by extensions.  An alternate solution is simply to reinforce the requirement for students themselves to report their results.  Focus group alumni indicated that they would have been happy to do so.   The information this provides may allow more effective oversight and also help to maintain links between employers and their staff.   Note that failure to provide results would also then constitute an indication of issues with individuals, and these may be acted upon by either the program or employers. 

Lesson 5: While flexibility is a highly valued aspect of the facility form of aid, if this flexibility is used in an ad-hoc manner, a loss of strategic direction and undesirable financial management burdens may result.
This lesson reflects earlier discussions regarding the lack of outcome-level objectives (and therefore strategic direction) offered by the design. It also refers to the complexities and administrative burdens placed on the AusAID initiative manager as a result of extensive ‘ad-hoc’ tasking of PATTAF by other AusAID Programs. 
That the country program as a whole found the flexibility of the facility very useful was clearly communicated by relevant respondents.  It would therefore be a short-sighted response to these issues to attempt to remove this flexibility completely.  The task of any new design for PATTAF is therefore to identify how much flexibility is still needed and design this into the facility such that strategic direction is not overly compromised and unintended administrative burdens are not unwittingly created.   That is, design for flexibility, rather than allow it to occur in an ad-hoc manner.
Much of the past tasking of PATTAF has come from the governance sector, but this demand has been reducing steadily in recent years.  With the advent of the Economic and Public Sector Program (EPSP) the demand from this area will likely fall away completely.  This means that the program can now return to a greater focus on its core business of scholarship provision, with fewer demands on the facility mechanism.  However, this does not mean that this need for all flexibility will disappear completely.  A new program will still need the ability to support small ancillary projects that are needed to effectively deliver scholarships, but are unable to be funded through a scholarship mechanism alone.  The clear example of this need that was provided by respondents to this mission was the supporting scholarship management information sharing between the program and Government stakeholders.  Without this support, engagement and integration with Government systems will remain highly problematic. It should also be noted that the EPSP is focused solely on the public sector, and any capacity-building initiatives arising which target the private sector may clearly still wish to utilise the flexibility offered by PATTAF model.  Also, while a number of higher education initiatives are in the pipeline, these are unlikely to be active for at least 12 to 18 months, and hence this may be another area in which a new version of a facility may be able to offer valuable interim flexible support.
Albeit significantly reduced, the need for this continuing flexible support will therefore require care to be taken to address strategic direction and administrative concerns when designing and delivering it.  More recent facility designs ensure strategic direction through the provision of clear guidelines and criteria regarding what it is permissible for them to support. The original PATTAF design did this to a limited extent, but any new design would need to be much more explicit in this respect.  Note that the intent of this guidance should not be to limit functional flexibility (i.e. how support is provided), but rather ensure that such flexibility is consistently used in a way that leads to effective contributions to desired program outcomes.  
Similarly, guidelines for administrative arrangements will need to be incorporated into any new program design.  These should require up-front agreement as to the allocation of roles and responsibilities of the managing contractor, the AusAID initiative manager for the facility and, most importantly, other programs or initiatives utilising the facility. Particular care should be taken to ensure that no party in unduly or disproportionately burdened by the arrangements.  Financial management arrangements and shared M&E requirements should be two key topics covered by such guidance.

Conclusions
In short, the program appears to have achieved what was required of it.  However, the outcome-level of achievement of components is either modest or difficult to determine due to design (and related M&E) constraints.  This apparently modest achievement is not necessarily an accurate reflection of impact. Stakeholder feedback consistently supports the view that significant benefits have flowed from the program. It is therefore difficult to give ratings of less than 4 in regard to most criteria. If such potential outcome-level benefits had been properly identified by the design and subsequently measured, this ICR could have been far more positive. However, in the absence of evidence of intentional outcome-level effects, it is equally difficult to provide ratings of above 4.
The staff of PATTAF appear to be of very high quality, especially in relation to their understanding of the PNG context and development of proposed reforms to current scholarship practices.  On the other hand, more could have been done by the program (even under the current design) to enable non-urban and other desirable groups to apply for, and eventually undertake scholarships.  GoPNG (and to some extent AusAID Canberra) have be reluctant to consider reforms that may have boosted program effectiveness, and AusAID Post have been slow to champion such reforms.
Without intending to ‘damn with faint praise’, an effective summary means of communicating the overall qualitative assessment of this ICR may be to simply make the observation that when faced with similar types and levels of constraints, AusAID scholarships/capacity-building programs have usually faired far worse than PATTAF.

Annex 1. Terms of Reference
TERMS OF REFERENCE
Independent Completion Report
PNG-Australia Targeted Training Facility (PATTAF)
 
1	Purpose

1.1	The purpose of independent evaluations of aid program activities is to provide information for AusAID’s assessment of aid program effectiveness. The Independent Completion Report (ICR) of the PNG-Australia Targeted Training Facility (PATTAF) and Australian Development Schoalrshsips (ADS) provided to PNG, will provide lessons to AusAID on program management and will also inform research, country strategies, design of new activities, and management of existing activities.

2	Background

2.1	PATTAF commenced in April 2002 and will operate until the proposed PNG Higher Education Development Facility design is implemented in March 2010.  The goal of PATTAF is to contribute to human resource development in areas of agreed focus in the PNG Development Cooperation Program. The purpose of the Facility is to provide targeted training in accord with the capacity and skills development needs of selected agencies.
2.2	Since it began, PATTAF has provided a broad range of services.  PATTAF has supported the private sector, provided Virtual Colombo Plan degrees and helped develop the Alumni Association for scholarship recipients. PATTAF also provides workforce development services for selected GoPNG Public Service Departments and manages AusAID’s tertiary study award programs (Scholarships) undertaken in Australia and the Pacific. These scholarships are primarily awarded under three AusAID funded programs: the Australian Development Scholarships (ADS), the Australian Leaderships Awards Scholarships (ALAS) and the Australian Regional Development Scholarships (ARDS).  
2.3	Selection of sectors for scholarship and workforce development activities are derived from an annual Targeting Strategy that identifies priority sectors within the PNG-Australia Development Cooperation Program (DCP). To ensure partnership and ownership, key Facility decisions are taken by the Facility Advisory Board (FAB) and Facility Coordination Group (FCG).  Membership of the FAB and FCG are determined in consultations between GOPNG and AusAID. 
2.4	Due to a range of issues and circumstance, the contract for the delivery of the services provided by PATTAF has been extended a number of times and the requirements of this service have been substantially modified as AusAID’s scholarship program and workforce development requirements have evolved. For example, changes to the requirements of service include temporarily filling a gap in the delivery of public sector short term training until the GoPNG Public Sector Workforce Development Program (PSWDP) and the Provincial Performance Initiative (PPII) were sufficiently strengthened to take on this responsibility. Responsibility has now been devolved to the PSWDP and PPII for complete GoPNG delivery. 
2.5	Over the life of the contract, the Facility has been identified by other AusAID programs as a suitable organisation to undertake program delivery services, including the Australian Leadership Awards Scholarships, the HIV and AIDS Leadership Support Initiative (LSI), the Strongim Gavman Program (SGP), the Career in Development Program (CiD) and the PNG Australia Alumni Association (PNGAAA). These programs operate under separate service orders and their success against intended objectives should be assessed and reviewed in the context of their respective programs, however, PATTAF’s effectiveness in responding to the evolving Human Resource Development priorities and development agendas should form part of this ICR.
2.6	The ADS program is the largest program currently being managed by PATTAF, requiring the selection and mobilisation to Australia of up to 130 awardees per year and the management of around 350 awardees who are ‘on-scholarships.’ The promotion, assessment and selection of ADS recipients is evenly split between Public and Open categories.  At the successful completion of their study program, alumni are expected to spend a minimum of two years in PNG and in this time contribute to PNG’s development outcomes.  
2.7	The ALAS program selects around seven to ten applicants per year for post graduate study in Australia from an open competitive application process. While the ARDS program is primarily managed from Suva, PATTAF manages the care of ARDS students from the Pacific who are undertaking study in PNG.
2.8	The impact of scholarship alumni in PNG is difficult to attribute and evaluate. One issue recently raised is a lack of clarity in scholarship objectives. Objectives have tended to be more high level 'motherhood statements' that are difficult to measure rather than statements of a tangible and measurable effect. Monitoring and evaluation should provide useful indicators and analysis that address both program outputs and outcomes. AusAID is committed to strengthening performance evaluation to help managers improve development effectiveness. It should be acknowledged that in the seven years since PATTAF commenced, AusAID’s thinking on M&E has developed considerably. The PATTAF ICR presents an opportunity to examine how M&E has evolved operationally, as each new phase of the contract has been rolled out.
2.9	In the current draft design for the new PNG Higher Education Development Facility, it has been proposed to encourage the use of 'profiles' to target scholarship promotion towards suitable candidates and the submission of 'reintegration' plans to be considered as part of the candidate selection process. These plans would form the basis of realistic and clearly enunciated scholarships objectives and therefore operate as a baseline for an evaluation of scholarships impact.

3	Objective of the ICR

3.1	The first objective will be to assess and rate PATTAF and the PNG ADS program against eight criteria: the five OECD/DAC criteria of relevance, effectiveness, efficiency, impact and sustainability, and the three additional AusAID criteria of monitoring and evaluation, gender equality and analysis and learning. The rating scale used is 1 – 6, with 6 indicating very high quality and 1 indicating very low quality. A rating below 4 indicates that an activity has been rated as less than satisfactory against a criterion. The evaluation team may draw on the attached evaluation questions (appendix A) to rate PATTAF against. 
3.2	The second objective is to focus on the monitoring and evaluation framework of PATTAF. In particular, the evaluation, addressing in particular the ADS scholarship program and  will address the following:
a)	Did the Program’s M&E Framework effectively address both program outcomes and outputs in respect of the ongoing collection of valid and useful data, the periodic evaluation of the program components and the broader evaluation of the program’s development outcomes? 
b)	Assess the effectiveness of the existing M&E framework as a tool to contribute to AusAID’s and the GoPNG’s corporate reporting and evaluation requirements.
c)	How rigorous was the framework’s application by the contractor and AusAID? 
d)	Were there areas of the program that were not adequately covered by the M&E framework in order for AusAID to fulfil its accountability, management and learning responsibilities
e)	Suggest a realistic M&E framework that collects qualitative and quantitative output and outcome data. 
f)	Consider how an M&E framework can contribute to building the capacity of GoPNG M&E systems inline with the spirit and purpose of the Accra Agenda for Action.
                   
4	ICR Review team composition
4.1	The PATTAF evaluation team will consist of: 
a)	an independent team leader with M&E specialisation, an understanding of development scholarship process and the political and cultural context of development scholarships in PNG;
b)	a second program evaluator from PNG; and
c)	a Canberra based AusAID staff member with development scholarship experience and experience in managing key relationships.

5	Roles and Responsibilities of the ICR team
5.1	The team leader will have overall responsibility for synthesizing evaluation material, production of the Evaluation Plan, draft and final ICR, and representation at in-Australia briefings. 
5.2	The program evaluator and team leader will be responsible for evaluation of various components of the PATTAF program in accordance with each member’s expertise and experience. The program evaluator will participate in in-country briefings in-Australia briefings (via video conference). The program evaluator will provide evaluation results to the Team Leader and liaise as needed with the Team Leader. 
5.3	The AusAID staff member will provide the team with insights into AusAID's policies and strategies relevant to the PATTAF ICR. They would provide support throughout the evaluation. 

6	Scope
6.1	There has recently been lengthy in-country mission consulting with many stakeholders within the GoPNG for the design of the new Higher Education Development Facility. Therefore a desk review of relevant documents (listed at appendix B) and the new Higher Education Development Facility designs documents should be completed prior to undertaking the ICR in-country mission.  It is however important to recognise the valuable insights partner agencies and departments can contribute to the ICR.
6.2	In PNG the ICR team will participate in consultations including:
a)	Briefing with AusAID in Port Moresby;
b)	Meetings with key interlocutors including the partner government representatives where required, the managing contractor and project beneficiaries; and
c)	Evaluation briefing with AusAID at the completion of the ICR.
6.3	In completing the first objective of the evaluation the mission will undertake focused case studies of selected projects within the Facility and assess the impact of their outcomes in meeting the goals of PATTAF. For example, focus groups may be held with returned alumnus to assess the outcomes and sustainability of the ADS. The LSI could be assessed through interviews with participants and organisation. 
6.4	In completing the second objective the evaluation will focus on the PATTAF’s monitoring and evaluation framework. Specifically the evaluation will:
a)	 consider lessons learned for strengthening performance orientation of facility based projects; and
b)	Provide strategic recommendations to inform the development of the M&E Framework for the new Higher Education Development Facility.

7	Duration
7.1	The evaluation is estimated to take up to 26 working days of the team leader’s time.  The research will require the ICR team members to be in PNG for five days. Including the feedback from AusAID, the evaluation will take eight weeks.
7.2	The research trip will require the ICR team members to be in PNG from 30 August to 07 September, 2009. A proposed timetable follows: 
Task	Location	Input (days)	
Document review	Home office	4	
Draft methodology	Home office	1	
AusAID Briefing.  Consultations with the Evaluation Officer, Higher Education and Training section	PNG	1 (+ travel)	
Evaluation mission – includes preparation and presentation of aide memoire	PNG	7 (+ travel)	
Draft  report	Home office	6	
Feedback from AusAID		(15)	
Participate and present at peer review	Canberra	2 (+ travel time)	
Final report	Home office	3	
TOTAL DAYS		26	

8	Output
8.1	The evaluation team shall submit the following outputs:
a)	an aide memoire at completion of the mission; 
b)	a draft report for consideration by AusAID within three weeks of completion of the field study to PNG to the Evaluation Officer, Performance Quality and Review Section, AusAID Canberra. Feedback from AusAID will be provided within two weeks of receiving the draft report; and 
c)	a final report for endorsement by AusAID and DNPM two weeks after feedback. 
8.2	The evaluation team will prepare a report of 25 pages maximum of text in accordance with AusAID’s Guidelines for IER reporting. The structure of reporting should be based on AusAID’s Guidelines for IER reporting as stipulated in AusAID’s ‘Rules and Tools’ for the ‘Completion and Evaluation of an Aid Activity’. (Guidance documentation to support the preparation of the IER will be provided by AusAID).  Lessons and recommendations should be clearly documented in the report.  

Appendix A

Questions for an Independent Completion Report

Relevance 
–	Were the objectives relevant to Australian Government and partner government priorities?
–	Were the objectives relevant to the context/needs of beneficiaries?
–	If not, what changes should have been made to the activity or its objectives to ensure continued relevance? 
Effectiveness 
–	Were the objectives achieved? If not, why?
–	To what extent did the activity contribute to achievement of objectives?
Efficiency
–	Did the implementation of the activity make effective use of time and resources to achieve the outcomes?
•	Was the activity designed for optimal value for money?
•	Have there been any financial variations to the activity? If so, was value for money considered in making these amendments?
•	Has management of the activity been responsive to changing needs?
•	Did the activity suffer from delays in implementation? If so, why and what was done about it?
•	Did the activity have sufficient and appropriate staffing resources?
•	Was a risk management approach applied to management of the activity (including anti-corruption)? 
•	What were the risks to achievement of objectives? Were the risks managed appropriately?
Impact (if feasible)
–	Did the activity produce intended or unintended changes in the lives of beneficiaries and their environment, directly or indirectly?
–	Were there positive or negative impacts from external factors?
Sustainability
–	Do beneficiaries and/or partner country stakeholders have sufficient ownership, capacity and resources to maintain the activity outcomes after Australian Government funding has ceased?
–	Are there any areas of the activity that are clearly not sustainable? What lessons can be learned from this?
Gender Equality
–	What were the outcomes of the activity for women and men, boys and girls?
–	Did the activity promote equal participation and benefits for women and men, boys and girls?
•	Did the activity promote more equal access by women and men to the benefits of the activity, and more broadly to resources, services and skills?
•	Did the activity promote equality of decision-making between women and men?
•	Did the initiative help to promote women’s rights?
•	Did the initiative help to develop capacity (donors, partner government, civil society, etc) to understand and promote gender equality?
Monitoring and Evaluation
–	Does evidence exist to show that objectives have been achieved?
–	Were there features of the M&E system that represented good practice and improved the quality of the evidence available? 
–	Was data gender-disaggregated to measure the outcomes of the activity on men, women, boys and girls?
–	Did the M&E system collect useful information on cross-cutting issues?
Analysis & Learning
–	How well was the design based on previous learning and analysis?
–	How well was learning from implementation and previous reviews (self-assessment and independent) integrated into the activity?
Lessons
–	What lessons from the activity can be applied to (select as appropriate: further implementation/designing the next phase of the activity/applying thematic practices [i.e. working in partner systems/environment/fragile stages] to the rest of the program/designing future activities). 

Appendix B

Reference Documents
•	AusAID Scholarship available on the internet: http://www.ausaid.gov.au/scholar/default.cfm and http://www.png.embassy.gov.au/pmsb/study.html
•	AusAID’s Guideline: Manage the Independent Evaluation of an Aid Activity
•	PNG-Australia Development Cooperation Strategy 2006-2010 (available from the internet)
•	PNG-Australia Partnership for Development (available from the internet)
•	Pacific Regional Aid Strategy 2004-2009 (available from the internet)
•	AusAID’s Australian Scholarships Group’s (ASG) Review of AusAID Scholarships Selection, Reintegration and Monitoring & Evaluation Processes 2008 (attached)
•	ASG’s Introductory Guidance on Monitoring and Evaluation for AusAID Study & Professional Development Award (SPDA) Programs
•	The original PATTAF design document and contract
•	The most recent contract and scope of service between AusAID and PATTAF

 Annex 2. Persons Consulted

Individuals/Groups	Organisation	
Phil Robson (Facility Director), Vivien Carol (Advisor), Berney Hirris (Advisor)	PATTAF	
Helen Koka, Linda Paru, Julie Yeta, Jeanette Renton, Joy Smara 	PNG Australia Alumni Inc – Female Focus Group	
George Taunakekei
Kaupa Awai, Valentine Tangoh, Varage John Laka	PNG Australia Alumni Inc –Male Focus Group	
Inaugural Exectives
-	Avia Koisen
-	Barrow Kirupana
-	Goodwill Amos
-	Helen Lesok

Current Executives
-	KellyCook
-	Avenama Rova
-	Nina Giheno
-	Lynna Justin
-	Thomas Willie
-	Janet Rangou
-	Pamela Mac’Givar
-	Menser Wagun
-	Edea Bouraga	PNG Australia Alumni Inc  - Executives:  Past and Present	
Mosiloyola Kwayaila	Former PATTAF FAB Board Chair	
Joseph Turia (First Assistance Secretary), Joelson Anere	Department of National Planning and Monitoring	
Phymbi Kokiva, Joan Quicho 	Department of Personnel Management	
Ritha Maken – HR Superviser	Oil Search Limited	
George Arua (Director), Thomas Kipau	National Training Council	
Linda Paru	Head of Human Resources - ANZ Bank	
Lindy Fisher	AusAID  - Port Moresby	
Warren Turner	AusAID  - Port Moresby   	
Donna-Jean Nicholson	AusAID  - Port Moresby 	
Bill Costello	Minister Counsellor – AusAID Port Moresby