International Monetary Fund

Independent Evaluation of the Pacific Financial Technical Assistance Center
(Volume 1 Main Text)

Bruce Murray (Team Leader) 
Richard Abrams
Kolone Vaai

22 May 2009

LIST OF ABBREVIATIONS

ADB 			Asian Development Bank
AFRITAC		African Technical Assistance Center
AFSPC		Association of Financial Supervisors of Pacific Countries
APD			Asia Pacific Department (IMF)
AML/CFT	Anti Money Laundering/Countering the Financing of Terrorism  
AUSAID 		Australian Agency for International Development
BOP			Balance of Payment
CARTAC		Caribbean Regional Technical Assistance Center
CPI 			Consumer Price Index
FAD 			Fiscal Affairs Department (IMF)
FEM			Forum Economic Ministers
FSAP	 		Financial Sector Assessment Program
FSS			Financial Sector Supervision
GDDS	 		General Data Dissemination System
GFS 			Government Finance Statistics 
IMF 			International Monetary Fund
MCM	 		Monetary and Capital Markets Department (IMF)
MFS			Monetary and Financial Statistics
MTEF	 		Medium-Term Expenditure Framework
MTFF			Medium-Term Financial Framework
NA			National Accounts
NSDS			National Strategy for the Development of Statistics
NZAID		New Zealand International Aid and Development Agency
OCO			Oceania Customs Organization
OTM 			Office of Technical Assistance Management (IMF)
PEFA			Public Expenditure and Financial Accountability
PFM  			Public Financial Management
PFTAC		Pacific Financial Technical Assistance Center
PICs			Pacific Island Countries
PIFMA 		Pacific Islands Financial Managers Association
PIFS			Pacific Islands Financial Soundness Indicators
PIN			Public Information Notice
PITAA			Pacific Islands Tax Administrators Association
RTAC	 		Regional Technical Assistance Center 
SDDS	 		Special Data Dissemination Standard
SNA 			System of National Accounts
SOE			State Owned Enterprises
SPC			Secretariat of the Pacific Community
STA 			Statistics Department (IMF)
TA			Technical Assistance
TAIMS		Technical Assistance Information Management System 
TPRC			Tripartite Review Committee
UNDP	 		United Nations Development Programme
USP			University of the South Pacific

Table of Contents
												Page
List of Abbreviations									   (i)
Table of Contents										  (ii)
Executive Summary										  (iv)

I.	Background and Introduction							     1
A.	Background									     2
B.	PFTAC Strategy and Objectives						     2
C.	Governance, Organization and Management of PFTAC			     3
D.	Focus of PFTAC Assistance							     5
E.	Financing PFTAC								     7
F.	Characteristics of the PFTAC Client Countries				     7
G.	Previous Evaluations of PFTAC						     9
H.	Evaluation Approach and Methodology					   10
II.	Assessment of PFTAC								   12
A.	Overall Assessment of PFTAC						   13
B.	Relevance of PFTAC								   14
C.	Effectiveness of the PFTAC Model						   16
D.	Efficiency of PFTAC								   21
E.	Sustainability of PFTAC							   25
F.	Tripartite Review Committee							   28
G.	Lessons for the Future								   29
III.	Assessment of Fiscal Assistance							   34
A.	Introduction to the Fiscal Assistance						   34
B.	Rating of the Fiscal Assistance						   37
C.	Relevance of the Fiscal Assistance						   38
D.	Effectiveness of the Fiscal Assistance					   40
E.	Efficiency of the Fiscal Assistance						   42
F.	Sustainability of the Fiscal Assistance					   44
G.	Lessons for the Future for Fiscal Assistance					   46
IV.	Assessment of the Financial Sector Supervision Assistance			   48
A.	Introduction to the Financial Sector Supervision Assistance			   49
B.	Rating the Financial Sector Supervision Assistance				   50
C.	Relevance of the Financial Sector Supervision Assistance			   51
D.	Effectiveness of the Financial Sector Supervision Assistance		   52
E.	Efficiency of the Financial Sector Supervision Assistance			   55
F.	Sustainability of the Financial Sector Supervision Assistance		   56
G.	Lessons for the Future for Financial Sector Supervision Assistance		   58
V.	Assessment of the Statistics Assistance						   59
A.	Introduction to the Statistics Assistance					   60
B.	Rating the Statistics Assistance						   61
C.	Relevance of the Statistics Assistance					   61
D.	Effectiveness of the Statistics Assistance					   63
E.	Efficiency of the Statistics Assistance					   67
F.	Sustainability of the Statistics Assistance					   68
G.	Lessons for the Future for the Statistics Assistance				  70
VI.	Recommendations									  72
A.	Implementation Status of 2004 Evaluation Recommendations		  73
B.	Recommendations								  73

List of Annexes (in Volume 2)
A.	Background Information on PFTAC
B.	Evaluation Approach and Methodology
C.	PFTAC Evaluation Survey Results
D.	Background Data and Survey Results for Fiscal Assistance
E.	Background Data and Survey Results for Financial Sector Supervision Assistance
F.	Background Data and Survey Results for the Statistics Assistance
G.	Implementation Status of the Recommendations in the 2004 Evaluation

Executive Summary

1.	PFTAC delivered high quality, effective and efficient services to its clients and its performance was rated as Good on a four point scale of Excellent, Good, Modest and Poor. Public Financial Management, Revenue Administration, Financial Sector Supervision and Statistics assistance were all rated as Good. Relevance, effectiveness and efficiency were all rated as Good. Concerns about sustainability led to a Modest rating. PFTAC is reliant on actions taken by the beneficiary agencies or factors beyond its control to achieve sustainability (e.g., institutional capacity issues; staffing and budgetary issues; resources from other donors to support the implementation of PFTAC’s recommendations). Because of limited institutional capacity, it sometimes takes several years and considerable support from other donors before recommendations are implemented. PFTAC is widely considered as having an excellent brand name and as the most effective regional technical assistance organization in the Pacific Region. 

2.	The Evaluation Team found that: (i) PFTAC’s assistance was “owned” by the countries and was responsive to their needs; (ii) the involvement of recipient countries, donors and IMF staff in the PFTAC governance structure was a successful model; (iii) recipient countries appreciated PFTAC’s provision of rapid, flexible services; (iv) the quality of PFTAC’s expertise was very good; (v) while all TA delivery modes were effective, the work of Resident Advisors was rated the highest; (vi) workshops were successful and participants used the knowledge gained on the job; (vii) and the Evaluation Team did not identify instances where PFTAC exposed IMF to reputational risk – on the contrary, PFTAC enhanced IMF’s reputation; (viii) PFTAC delivered TA to a large number of small countries – delivering the same volume of TA from Washington would not be feasible; (ix) PFTAC supported regional initiatives; (x) backstopping from Headquarters contributed to PFTAC’s effectiveness and good quality of assistance but the system is under stress; and (xi) PFTAC made limited use of Pacific expertise. The two most frequently cited factors contributing to PFTAC’s success were its location in the region and its high quality of expertise. Other reasons included PFTAC’s responsiveness to government priorities, consistent engagement over a period of years, personal relationships with its clients, knowledge of its client countries and good coordination with TA providers and IMF Headquarters.

3.	The most pressing area for improvement is enhancing the sustainability of the benefits of PFTAC’s assistance. This will involve more follow up to help implement recommendations, proactively searching for synergies with other donors to support capacity building and more support for regional initiatives. There were a number of excellent partnerships with other donors. Other donors provided substantial assistance to support implementation and build capacity and PFTAC helped to identify priorities, scope out the assistance and sometimes peer reviewed the work. This excellent model should be used more frequently in the future. Other areas for improvement included more use of short term experts, greater use of Pacific expertise, better monitoring TA outcomes and improving some IMF policies and procedures governing RTACs. 

4.	Because of its modest size, PTFAC cannot meet the demands for its services or have the impact described in the Project Document. Building capacity, as opposed to providing short term on demand advice, requires a deeper engagement. Stakeholders need to consider whether they wish PFTAC to continue operating in the same way that it has since it was established 1993 or whether to scale up a successful model. Doing so would require a significant increase in PFTAC’s resources. If the stakeholders do not provide the necessary resources, the objectives in the next Project Document should be scaled back to be more consistent with the available resource envelop. The recommendations were focused on issues that need to be considered if the PFTAC model is to be elevated to the level. Eight recommendations are proposed to provide input for the drafting of the Project Document for the next funding cycle.

•	Recommendation 1: By the end calendar year 2010, TPRC and IMF should develop a strategic plan that sets out a vision for where PFTAC should be in five years time. Scaling up the successful PFTAC model would involve increasing the number of resident advisors (see Recommendation 2), making greater use of short term experts (Recommendation 3), increasing the chance that the benefits of PFTAC’s assistance will be sustainable (Recommendation 4), making greater use of Pacific expertise (Recommendation 5) and intensifying the use of regional approaches (Recommendation 6). Broad agreement on the way forward should be reached at the June 2009 TPRC meeting.

•	Recommendation 2: By the end of FY2010, consensus should be reached at TPRC on the priorities for additional Resident Advisors. The Evaluation Team identified three priorities: (i) Macroeconomic Advisor; (ii) second PFM Advisor; and (iii) second Statistics Advisor.

•	Recommendation 3: Beginning in FY2010 PFTAC should make greater use of short term experts to leverage the expertise and associated fixed costs of Resident Advisors, particularly in the financial sector supervision and statistics areas. 

•	Recommendation 4: By the end of FY2010 the TPRC, PFTAC Coordinator, Resident Advisors and the TA Departments should develop a strategy to strengthen the likely sustainability of the benefits of PFTAC assistance.  

•	Recommendation 5: Beginning in FY2010 PFTAC should make a concerted effort to develop and use Pacific expertise.

•	Recommendation 6: By the end of FY2011 TPRC and PFTAC should develop a strategy to intensify and extend the use of regional approaches to build capacity in the Pacific Region.

•	Recommendation 7: By the end of FY2011 PFTAC should, in coordination with the TA Departments, clearly define medium term objectives to be achieved in each functional area in each country and verifiable indicators against which progress can be monitored.

•	Recommendation 8: By the end FY2010 PFTAC should develop formal action plans for each recommendation accepted by the TPRC, identifying the necessary resources and monitorable benchmarks to implement those recommendations and report on the implementation status of the action plans to the TPRC in FY2011 and again in FY2012.
•	
I.	BACKGROUND AND INTRODUCTION

Key Messages

•	PFTAC’s objective is to help build institutional and human capacity in public financial management, revenue administration, financial sector supervision and statistics in the Pacific Region. 
•	PFTAC has 15 client countries and operates in a challenging operating environment. Many clients are small and have limited institutional capacity. 
•	Although the funding has increased, the structure of PFTAC and the range of services it delivers have remained constant since it was established in 1993.
•	All three previous evaluations concluded that PFTAC was successful but it cannot meet the demand for its services because of capacity limitation. 
•	A quantitative rating methodology was applied in an evaluation framework based on relevance, effectiveness, efficiency and sustainability. Evaluation findings were aggregated across functional areas and to assess PFTAC’s overall performance. 	

A.	Background

1.	The Pacific Financial Technical Assistance Center (PFTAC) was established in Suva, Fiji in 1993 to provide technical advice and capacity building in economic and financial management to 15 Pacific island countries (PICs). Initially PFTAC was jointly established by the United Nations Development Programme (UNDP) and the International Monetary Fund (IMF). In 2002, UNDP ceased supporting PFTAC as a donor because of funding constraints and changes in priority. Since then IMF has mobilized funding from key partners to finance PFTAC.

2.	This evaluation primarily covers the period from FY2006 to FY2008. The Project Documents for FY2006/08 and FY2009/11 set out PFTAC’s strategy and objectives. The broad strategies and objectives in the two documents are similar. Changes reflect differences in detail and incremental improvements rather than major changes in direction. However, there have been substantial increases in the amount contributed for PFTAC. The $8.3 million for the FY2006/08 funding cycle represented a 44% increase over the amount contributed for the FY2003/05 period. There was a further 18% increase and funding for FY2009/11 reached nearly $10 million (see Table A.1). 

B.	PFTAC Strategy and Objectives

3.	The FY2006/08 Project Document indicates that policy weaknesses adversely affected the economic performance of some PICs. PFTAC seeks to help PICs to design reforms and build capacity that lead to: 
(i)	simple and efficient revenue instruments and effective tax administrations capable of increasing tax compliance, and modernized customs procedures to secure revenue and facilitate trade; 
(ii)	efficient, effective, transparent and sustainable budget formulation and presentation, budget execution and control, reporting and audit; 
(iii)	compliance with international standards and best practices in prudential supervision and regulation, including measures to deter and detect money laundering and terrorism financing; and, 
(iv)	regular and timely compilation, analysis and dissemination of economic and financial statistics according to accepted international standards.

4.	PFTAC is one of the six centers established within the framework of IMF’s Regional Technical Assistance Center (RTAC) policy. The creation of the RTACs was viewed as an opportunity to improve the effectiveness of IMF TAs and to compliment a strategic shift  in Headquarters delivered TA to “upstream” or “strategic” TA and to better integrate TA with the IMF surveillance and program activities. The RTACs were also designed to improve the coordination of IMF’s TAs with other TA providers and to promote country ownership of reforms. Backstopping from Headquarters provides quality assurance. RTACs have become an increasingly important way for IMF to deliver TA.

5.	In 2008 IMF adopted a new policy to enhance the impact of TA. The key features of that policy include: (i) better integrating TA and IMF surveillance and lending operations; (ii) improving the prioritization of TAs by better aligning TA with strategic objectives of countries; (iii) better integrating TA into IMF’s medium-term budget; (iv) using performance indicators to make TA more transparent and accountable – TAs will have clear objectives and deliverables against which progress will be measured and TA evaluation will become more systematic in monitoring and assessing results and better disseminate lessons learned; (v) improving the costing of TAs; and (vi) strengthening IMF’s partnerships with donors. These initiatives should have a positive impact on PFTAC going forward.

C.	Governance, Organization and Management of PFTAC

6.	PFTAC is operated by the IMF in consultation with the PIC governments and the development partners supporting it (see Figure I.1). The organization chart is quite complex and shows four direct and nine indirect reporting lines. This suggests significant organizational coordination costs and possibly bottlenecks and inefficiencies in the flow of information between the parties involved. In the medium term, in the context of the next RTAC review, consideration should be given to streamlining the reporting lines and relationships.

7.	The Tripartite Review Committee (TPRC) provides strategic direction and guidance within the framework of the Project Documents. The Governor of the Reserve Bank of Fiji is the TPRC chairman. TPRC provides beneficiary countries and contributing donors with a role in the formal decision making process.

8.	IMF is responsible for PFTAC’s managerial, technical and administrative arrangements and the relevance and quality of PFTAC assistance. IMF: (i) assigns the Center Coordinator; (ii) selects the Resident Advisors and hires short-term experts; (iii) provides supervision and backstopping services; (iv) provides administrative support; and, (v) manages the cost-sharing contributions of donors and the financial operations of PFTAC through the Office of Technical Assistance Management (OTM). OTM also develops IMF’s broader policies for TA management, maintains relations with donors, mobilizes TA funds, broadly monitors their use, manages the finances and commissions independent evaluations of TA activities.

9.	The PFTAC Coordinator is a full time IMF staff who is appointed by, and reports to, the Asia and Pacific Department (APD). The PFTAC is the TPRC secretary and is responsible for the day-to-day management, preparing and executing PFTAC’s work program and liaising with bilateral and multilateral funding agencies to ensure effective coordination and complementarity among TA programs.

10.	There are four Resident Advisors with expertise in: (i) public financial management (PFM); (ii) revenue administration; (iii) financial sector supervision (FSS); and (iv) statistics. The Resident Advisors are identified from IMF’s panels of experts and are selected, and appointed by the concerned TA department. The number of Resident Advisors and their skill mix has not changed since PFTAC was established. The Resident Advisors are appointed on one year contracts, which can be extended. The procedures for extension are not uniform across the three TA departments. The Resident Advisor vacancies are not routinely publically advertised. While some positions were advertised internationally, others were only advertised on the IMF webpage, and some were not advertised. Some TA Departments prefer to select Resident Advisors from their rosters of vetted experts and only advertise in cases where difficulty is experienced in finding suitable and willing candidates. To improve transparency, the Evaluation Team believes that all Resident Advisor positions should be publicly advertised as a matter of policy. Although footnote 9 in the January 2006 RTAC Operational Guidance Note states that Resident Advisors may be IMF staff or external consultants, in practice IMF staff wishing to work in an RTAC must go on Leave Without Pay. In addition to the direct supervision by the TA departments, the PFTAC Coordinator, PFTAC office staff, OTM and IMF’s information technology and systems and business, travel and human resource processes can impact on the performance of the Resident Advisors and PFTAC more generally.

Figure I.1 PFTAC Organization Chart

TA Department (MCM, FAD, STA)

RESIDENT ADVISORS

•	Public Financial Management
•	Revenue/Customs Administration
•	Financial Sector Supervision
•	Statistics

PFTAC COORDINATOR

Tripartite Review Committee

TA Recipients

Headquarters Technical Assistance

PFTAC Work Plan

Asia and Pacific
Department

endorses

Office of Technical Assistance Management

IMF Regional
Plans

informs

11.	The need for short-term experts depends on the PFTAC work plan and available resources. Short-term experts are normally recruited and supervised by the PFTAC Advisors. To guarantee the quality of the TA delivered, all short-term experts undergo a certification process by the relevant TA department. There was some concern in IMF that channeling TA through RTACs might adversely affect IMF’s reputation. To reduce this risk and ensure high quality advice and projects, Resident Advisors are backstopped by the TA departments and short term experts are backstopped by the TA departments and/or the Resident Advisors.

D.	Focus of PFTAC Assistance

12.	PFTAC’s work plans are to: (i) reflect the needs of member countries; (ii) be integrated with the TA, surveillance, and lending activities of IMF Headquarters; and (iii) be coordinated with TA provided by other donors. PFTAC provides assistance at both the national and regional levels. At the national level, which was to receive priority in the work plan, PFTAC responds to formal requests from PICs. Priorities for regional activities reflect guidance provided by the TPRC and discussions at Forum Economic Ministers (FEM) meetings. Regional activities address common issues and challenges faced by PICs where the potential benefits from a regional approach are evident. PFTAC’s mission, vision and values statement is given in Box I.1. 

Box I.1: PFTAC’s Mission, Vision and Values Statement	

•	PFTAC’s Mission: To enhance the institutional and human capacities of member countries and regional bodies in the Pacific region to achieve their financial and economic policy objectives.
•	PFTAC’s Vision: To be a center of quality advice and capacity building on the technical aspects of financial and economic policy formulation and management to member countries, regional bodies and other TA providers in the Pacific region.
•	PFTAC’s Values: PFTAC will seek to adhere to five values in pursuit of its mission: 
•	Integrity in its policy advice and dealings with member countries, regional bodies and other TA providers;
•	Responsive to the needs of member countries and regional bodies;
•	Focus on efforts to tailor policy advice and capacity building efforts to the needs and capacities in the Pacific region;
•	Quality of advice will be maintained by careful selection of experts by the IMF and  by backstopping services provided by functional departments in Washington; and,
•	Cooperation with other TA providers and regional institutions to ensure coordinated responses to member countries.
	
Source: Project Document FY2006 to FY2008. Page 10.	

13.	This evaluation focuses on the assistance financed from PFTAC’s budget and does not cover TA delivered from IMF Headquarters. PFTAC uses a number of modalities to deliver its assistance, including (i) missions by Resident Advisors (131 or about one mission per month per Resident Advisor); (ii) missions undertaken by short term experts (55); (iii) training provided through regional seminars (280 participants) and attachments (34); (iv) provision of support for the three regional associations; (v) funding agency coordination and consultation with TA design and peer review of the resulting material; (vi) facilitating requests for TA from the PICs and identification of suitable funding agency partners; and (vii) acting as a regional resource center for information in PFTAC’s areas of specialization. From FY2006/09 PFTAC provided 172 person months of input from Resident Advisors and short term experts – 78% by Resident Advisors and 22% by short term experts. The fiscal area accounted for 60% of the inputs, financial sector supervision for 21% and statistics for 19%. Extensive use was made of short term experts in the fiscal area (34 person months). No short term experts were used for financial sector supervision and 1.5 person months was used for statistics. However this underestimates IMF’s support provided in the financial supervision and statistics areas. For the former some short term experts were provided by MCM and for the latter additional TA was mobilized by the Statistics Advisor, which was administered by STA.

14.	Because of disparities in the size and structure of their economies, PFTAC client countries are not homogeneous and the nature and level of assistance needed varies considerably in the region. For example the needs of the larger countries [e.g., Fiji; Papua New Guinea (PNG)] are different from those of the smaller PICs. It is not surprising that PFTAC’s assistance was not evenly distributed across the PICs. The top six countries in terms of amount of PFTAC assistance received accounted for 53% of the combined missions of Resident Advisors and Short Term Experts while the bottom five countries received 16% of the missions. Relative to the size of their economies, Palau and the Federated States of Micronesia (FSM) received a large share of missions and PNG and Samoa received proportionately less. The 280 workshop participants were dominated by people from PNG, Vanuatu, the Cook Islands, Fiji and Samoa while only 14% came from the smallest countries. The 34 attachments were dominated by officers from Vanuatu (7), FSM (6), PNG (4), Tonga (4) and Fiji (3). Several countries received either one or no attachments during the evaluation period. 

15.	It is not possible for each Resident Advisor to travel to all 15 PFTAC member countries every year. Doing so would dilute the effectiveness of TA and few results would be achieved. Choices must be made in formulating PFTAC’s work program which effectively means that some countries will not receive any assistance from PFTAC in a particular functional area for one or two years. At a practical level PFTAC considers the following factors when prioritizing requests for assistance: (i) giving priority to countries which have a solid reform program; (ii) giving priority to countries that have a track record of implementing the recommendations of previous assistance; (iii) responding to countries with specific and immediate needs; (iv) ensuring that small countries receive some assistance; and (v) supporting regional initiatives that benefit all, or most, PICs. Given that one of PFTAC’s strengths is that it responds rapidly to important ad hoc requests for assistance, it is inevitable that there will not be clear strategic patterns for PFTAC’s assistance. Given PFTAC’s operational constraints, the geographic distribution of its assistance appears to be reasonable.

E.	Financing PFTAC

16.	During the evaluation period, PFTAC was financed by the Australia, New Zealand and the Asian Development Bank though contributions to the PFTAC Sub Account and project/activity funding from Japan and Korea. IMF finances the cost of the PFTAC Coordinator, most local staff costs and office costs. Figures are not available as to the cost of indirect IMF support for the PFTAC. Beneficiary countries do not contribute to PFTAC’s financing.

17.	Financial support for PFTAC has increased substantially over the years. Donor contributions for the FY2006/08 funding cycle totaled $8.3 million. ADB provided 10% of the financing; Australia, 16%; New Zealand, 23%; Korea, 5%; Japan, 24%; and IMF, 22%. The main changes in the funding sources during the FY2006/08 funding cycle compared to FY2003/05 were: (i) a particularly generous increase in New Zealand’s contribution; (ii) a significant increase by IMF; (iii) Korea’s participation as a new contributor; and (iv) a fall in ADB’s contribution. For the FY2009/11 period, there were significant increases in the contributions from Australia and Korea and increases from ADB, Japan and New Zealand. IMF’s contribution fell.  New Zealand, Australia and Japan are the three largest sources of financing for PFTAC, each contributing over $2 million for the current funding cycle (see Table A.1 in Annex A). Annual PFTAC expenditures were $2.17 million in FY2006, $2.95 million in FY2007 and $2.68 million in FY2008 (see Table A.2).
  
F.	Characteristics of the PFTAC Client Countries

18.	The characteristics of the PFTAC client countries have implications for the nature of the needs and the delivery modes (see Table A.3). Key characteristics include: (i) the 15 PICs cover a vast area in the Pacific Ocean; (ii) air travel is time consuming and expensive and flights to some countries are infrequent; (iii) long distances from major markets; (iv) with the exception of PNG, all PICs have populations of less than one million and nine have populations of 100,000 or less; (v) with the exception of PNG, the land area in all countries is less than 30,000 sq km and is less than 1,000 sq km in 10 countries, (vi) with the exception of PNG and Fiji, total GDP is less than $1 billion and less than $500 million for eight countries; (vii) GDP per capita ranges from a low of $1,800/$1,900 for PNG and the Solomon Islands to a high of $14,300 for Palau; (viii) the size and scale of the institutions are commensurate with the small size of the countries – the number of skilled, well qualified professionals is limited and the loss of a few trained, experienced staff can have a major adverse impact on institutional capacity, (ix) since the turn of the Millennium, most Pacific economies have grown by 2% to 3% per year,  (xi) since 2000 inflation has been under control in most PICs, although it exceeded 5% per year in PNG, Samoa, Solomon Islands and Tonga; (xi) the revenue to GDP ratio exceeded 20% in all countries; (xii) small economies are vulnerable to external shocks (e.g., the economic fallout from ongoing global financial crises; weather related events like cyclones); and (xiii) many PICs are heavily dependent on donor assistance. 

19.	Five PFTAC members are not members of IMF (e.g., Cook Islands; Nauru; Niue; Tokelau; Tuvalu). These countries receive TA from IMF Headquarters only in exceptional circumstances and do not benefit from Article IV consultations. Six PFTAC members have central banks (e.g., Fiji; PNG; Samoa; Solomon Islands; Tonga; Vanuatu), their own currencies and independent monetary policies. The other countries use other currencies, typically the American, Australian or New Zealand dollar. This limits the role of monetary policy. Some of these countries have a financial supervisory body. 

20.	Country characteristics influence the capacity of institutions to successfully absorb TAs and for TAs to achieve effective outcomes that are sustainable. TAs are more likely to be successful in countries that have political and institutional stability, macroeconomic stability, and sound economic management. Institutions that are adequately staffed with capable people are more likely to be able to work as full partners with Resident Advisors or short term experts in implementing TAs. If these characteristics are absent in various degrees, there is an increased risk of Assistance not achieving the desired results. For this evaluation, two of the World Bank’s six governance indicators were used as proxies for general institutional absorptive capacity:  
(i)	Government Effectiveness, which measures perceptions of the quality of public services, the quality of the civil service and the degree of independence from political pressures, the quality of policy formulation and implementation, and the credibility of the government's commitment to policies; and,
(ii)	Regulatory Quality, which measures perceptions of the ability of the government to formulate and implement sound policies and regulations that permit and promote private sector development.

21.	The data in Table A.4 shows a mixed picture in terms of government effectiveness and regulatory quality in the PICs. The institutional environment in some countries in which PFTAC operates is more challenging than in others. The institutional absorptive capacity was rated as Good for two countries (e.g., Cook Islands; Samoa), Modest for three PICs (e.g., Fiji; Vanuatu; Tuvalu) and as Challenging in six countries (e.g., Kiribati; Marshall Islands; Nauru; PNG; Solomon Islands and Tonga). Data was not available for four countries (e.g., FSM; Niue; Palau; Tokelau). One other governance indicator sometimes affects the absorptive capacity relates to political stability and the absence of violence. With the exception of PNG, PICs scored relatively well on this criteria.

22.	All major donors devote a substantial portion of their assistance programs in the Pacific to capacity building. Based on the data in Table A.4, there is no evidence that government effectiveness and regulatory quality in the PICs have improved significantly since 2000. The only countries in which there was a change that was statistically significant over the seven year period scored worse in 2007 than in 2000. This raises general questions about the likelihood of the effectiveness and sustainability of capacity building assistance and indicates that PFTAC is operating in a challenging environment. 

G.	Previous Evaluations of PFTAC

23.	IMF has commissioned two previous independent evaluations of PFTAC in 2004 and over a decade ago in 1997. ADB evaluated PFTAC in 2006. The main conclusions of the 2004 evaluation were: (i) PFTAC played a positive role in the region and was appreciated by the participating countries, regional bodies, donor agencies and other stakeholders; (ii) PFTAC filled a niche that cannot be filled by other TA delivery modalities; (iii) delivering technical assistance by Resident Advisors based in the region worked and was superior to the alternative of trying to provide all TA for the PICs out of IMF Headquarters; (iv) PFTAC was an appropriate model; (v) while the quality and timeliness of PFTAC’s work was high, the volume of PFTAC’s output and the access to its services were limited by resource constraints; (vi) PFTAC should leverage its resources by accommodating a tier of locally recruited experts and using more short term experts; (vii) PFTAC’s high quality staff resources contributed to its good performance; (viii) PFTAC interventions were most successful when there was strong country ownership; (ix) greater efforts are needed to monitor and measure outcomes of PFTAC’s interventions; (x) PFTAC  exercises considerable autonomy in its decisions on how best to serve PICs; (xi) TPRC has been effective as PFTAC’s principal governance body; and (xii) lack of capacity and sustaining existing capacity are major challenges in PICs. 

24.	The findings of the 1997 evaluation were also generally positive: (i)  PFTAC was well conceived, was the right model, had many advantages over other TA modalities and the main features of the PFTAC model should be retained; (iii) demand for PFTAC assistance has exceeded PFTAC’s capacity to deliver; (iv) several PICs have narrow human resource bases; (v) the regional approach to delivering TAs has many notable benefits; (vi) the high caliber of staff contributed to PFTAC’s success; (vii) the system for monitoring and evaluating TAs was not effective; and (viii) given the high quality assistance, PFTAC was relatively cost effective.

25.	The main conclusions of ADB’s evaluation were: (i) PFTAC was rated as successful; (ii) PFTAC’s bank supervision work was rated as highly successful, PMF and revenue/customs administration TAs were rated as successful and statistics TAs were rated as partly successful; (iii) PFTAC objectives were highly relevant and the anticipated gains from a regional design were realized; (iv) PFTAC did not provide TA for the development of macroeconomic capability, a weakness in its scope of services – PICs need help to prepare economic forecasts, macroeconomic frameworks and financial programs; (v) PFTAC was assessed as effective, although the lack of available performance measures to monitor its activities made rigorous assessment problematic; (v) PFTAC operations were rated as efficient, reflecting the regional approach in service delivery; (vi) efficiency gains and a reduction in unit costs could be achieved by increasing the use of short-term, experts; (vii) while capacity building in the Pacific region is known to be difficult due to the narrow human resource base and ease of migration, capacity building has progressed in most areas of PFTAC activities; (viii) PFTAC was likely to be sustained; and (ix) ADB should have closer cooperation with PFTAC during its country programming.

H.	Evaluation Approach and Methodology

26.	Undertaking an independent evaluation is part of PFTAC’s governance structure. The evaluation had four objectives: (i) evaluate the TA provided by PFTAC for relevance, effectiveness, efficiency and sustainability; (ii) assess whether PFTAC is the right size to achieve its objectives, or alternatively, if the objectives need to be adjusted; (iii) examine the effectiveness and frequency of TPRC meetings, and explore alternate modalities for more frequent contact; and (iv) compile a set of lessons that may be used to strengthen PFTAC operations.

27.	The evaluation approach and methodology are described in Annex B. The terms of reference required the application of a quantitative rating methodology within an evaluation framework that is based on relevance, effectiveness, efficiency and sustainability. Each cluster of assistance, public financial management, revenue/customs administration, financial sector supervision and statistics, was rated against the specific sub-criteria for the four dimensions of evaluation and aggregated using assigned weights. A four point rating scale was used – excellent, good, modest and poor.

28.	The evaluation drew on information from documents and data available from IMF, interviews at IMF Headquarters and with TRPC members, the PFTAC Coordinators and Resident Advisors, government officials, workshop participants and representatives of other TA providers. The Evaluation Team visited three of the 15 countries serviced by PFTAC. To broaden the coverage of the evaluation an electronic survey was undertaken of key informants. A total of 569 survey questionnaires were successfully delivered and 266 responses were received when the results were down loaded for this report, equivalent to a 47% response rate. There were reasonable number of responses from government agencies (central banks, ministries of finance, statistical agencies), IMF staff and short term experts and donors and reasonable geographic coverage. Nearly 30% of the survey respondents said that they were very familiar with PFTAC, 40% were familiar with PFTAC and an equal number, about 15%, said that they were somewhat familiar with PFTAC or had had only limited interaction with PFTAC (see Tables C.1 to C7 in Appendix C). 

29.	The Evaluation Team is aware of the methodological challenges associated with evaluating TA for capacity building. Those challenges were addressed by using triangulation but the information base was incomplete and a considerable amount of judgment was applied. The evaluation methodology was designed to make those judgments transparent to readers. The evaluation relied, to a considerable extent, on perceptional data, i. e., opinions, views and comments made by various key informants and survey respondents. In making judgments, the Evaluation Team considered evidence from several sources to validate key conclusions.

II. ASSESMENT OF PFTAC

Key Messages

•	PFTAC’s overall performance and its PFM, revenue administration, financial sector supervision and statistics were all rated as Good. 
•	Relevance, effectiveness and efficiency were rated as Good. However, because of limited institutional capacity, it often takes several years and support from other donors before recommendations are implemented.
•	Concerns about sustainability led to a Modest rating. Institutional weaknesses, a lack of follow up and limited resources to support the implementation of recommendations undermined sustainability. 
•	The PFTAC model is consistent with the principles in the Paris Declaration and Accra Agenda for Action and has been durable, successful and cost effective in delivering high quality services in the Pacific Region. It is arguably the most successful regional organization in the Pacific and fulfills a niche that cannot be filled by other TA delivery modes.
•	The Resident Advisors are over stretched and PFTAC, with its modest staffing levels, cannot meet the demand for its services. 
•	The Evaluation Team identified a number of successful partnerships with other donors. PFTAC identified priorities, helped other donors scope out the assistance and peer reviewed the resulting work. Opportunities should be sought to use this successful model more frequently in the future.
•	The two most frequently cited factors contributing to PFTAC’s success were its location in the region and the high quality of its expertise. Other factors contributing to its effectiveness included its responsiveness to government priorities, consistent engagement over a period of years, personal relationships built up with its clients, knowledge of its client countries and good coordination with TA providers and Headquarters.
•	The regional workshops were well organized and participants used the knowledge gained on the job.
•	Areas for improvement include more follow up to help implement recommendations, more use of short term experts, more support for regional initiatives, greater use of Pacific expertise and better monitoring and evaluation of TA outcomes.
•	There are no significant differences across the functional areas in the cost of delivering one person month of input or between PFTAC and other RTACs.
•	Some IMF policies and procedures governing RTACs need improvement. 	

A.	Overall Assessment of PFTAC

30.	This chapter assesses the general features of the PFTAC model and its overall performance. This evaluation confirmed the findings of the previous evaluations that PFTAC has delivered high quality, effective and efficient services. PFTAC’s overall performance was rated as Good as was the assistance in all four functional areas -- PFM, revenue administration, financial sector supervision and statistics (see Table II.1). Relevance, effectiveness and efficiency were rated as Good. Concerns about sustainability led to a Modest rating. Sustainability was consistently ranked the lowest across all clusters of PFTAC’s assistance. PFTAC is reliant on actions taken by the beneficiary agencies or factors beyond its control to achieve high scores for sustainability (e.g., broader institutional capacity issues, including staffing and budgetary issues in the beneficiary agencies; resources from other donors to support the implementation of PFTAC’s recommendations). Experience indicates that because of limited institutional capacity, it sometimes takes several years and considerable support from other donors before PFTAC’s recommendations are effectively implemented.

Table II.1: Rating PFTAC by Functional Area
(Note: This information replaces a table that is available in the PDF version of this report)
Input of Resident Advisors (Person Months)
Weight: N/A per cent; PFM: 36; Revenue Administration: 32; Financial sector supervision: 36; Statistics: 32.5; Total: 136.5
Relevance
Weight: 20 per cent; PFM: 3.0; Revenue Administration: 3.0; Financial sector supervision: 3.1; Statistics: 2.5; Total: 2.9
Effectiveness
Weight: 40 per cent; PFM: 3.2; Revenue Administration: 3.2; Financial sector supervision: 3.2; Statistics: 3.0; Total: 3.2
Efficiency
Weight: 20 per cent; PFM: 3.6; Revenue Administration: 3.2; Financial sector supervision: 3.2; Statistics: 3.2; Total: 3.3
Sustainability
Weight: 20 per cent; PFM: 2.4; Revenue Administration: 2.4; Financial sector supervision: 2.8; Statistics: 2.0; Total: 2.4
Total rating
Weight: 100 per cent; PFM: 3.1; Revenue Administration: 3.0; Financial sector supervision: 3.1; Statistics: 2.7; Total: 3.0
Note: Column weights were defined by the Evaluation Team and row weights are based on the proportion of person months of Resident Advisor and Short Term Expert input for each group of TA activities.
Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

31.	PFTAC is widely considered to be the most effective regional organization in the Pacific Region. Consistent with the tone of the feedback received by the Evaluation Team, two survey respondents wrote: "PFTAC needs more support from IMF headquarters to sustain the great work it has done over the years. The value of this work is immeasurable and the benefits that accrue for this assistance will sustain regional island countries for many years to come. Thank you PFTAC.” and “In general I consider PFTAC provides greater support and technical assistance than would be suggested by its size. Flexibility and responsiveness is key to this, as is the development of relationships with beneficiary countries. PFTAC is in an unenviable position where sustainable development initiatives require small, unhurried and iterative approaches over many years. Yet these small developments may not be recognized by contributing donors as real achievements when measured against their evaluation frameworks. When balancing the competing and divergent needs of donors and beneficiary countries I consider PFTAC does an admirable job.”

32.	One issue identified in the 2005 RTAC review, was the question of who had the authority to resolve differences between country driven RTAC work programs and the priorities of the IMF’s regional and TA departments. This issue was thought to have the potential to create reputational risks for IMF. The Evaluation Team did not come across any instances where PFTAC’s work resulted in serious reputational risks for IMF. The feedback was quite the reverse – the work of PFTAC enhanced IMF’s reputation. 

B.	Relevance of PFTAC

1.	Overall Assessment of Relevance

33.	The relevancy of PFTAC’s assistance was rated as Good (see Table II.1). There is an extensive literature that demonstrates that institutions matter for economic development and for the formulation and implementation of sound public policies. The Pacific strategies of the Asian Development Bank (ADB), the Australian Agency for International Development (AUSAID) and the New Zealand Aid and Development Agency (NZAID) all stress the importance of capacity building and sound macroeconomic management, particularly in the fiscal area. Most observers agree that the lack of technical skills and institutional capacity to formulate and implement appropriate economic and financial policies has constrained development and has contributed to disappointing economic progress in some countries. The analysis of major donors operating in the Pacific is consistent with PFTAC’s focus on building macroeconomic management capacity. 

34.	PFTAC’s assistance was consistent with government priorities and government owned. This was confirmed by the results of the PFTAC Evaluation Survey and in field interviews with senior officials. About half of the respondents assigned a Good rating for the consistency of PFTAC assistance and government priorities. Nearly 40% rated it as Excellent and very few rated it as Modest or Poor. These patterns were broadly similar regardless of employer or whether government officials worked in ministries of finance, revenue administration, central banks or statistics agencies (see Tables C.9 to C.12). Given these findings, the Evaluation Team concludes that PFTAC has successfully addressed two issues identified in the Project Document: (i) increasing government ownership of, and commitment to, TAs; and (ii) better designing TAs to reflect local needs and implementation capacities.

35.	PFTAC played a role in helping PICs to define their TA priorities. Respondents were nearly evenly split about whether PFTAC played a role but not leading role in defining TA priorities or whether PFTAC played an important and leading role with about 45% in each category. Very few replied that PFTAC made minor inputs or did not play any role (see Table C.13). These results were broadly consistent with the feedback received from the authorities interviewed. Generally the countries identified the broad areas in which they needed assistance and PFTAC helped to set priorities within that area.

2.	Strategic Relevance of PFTAC

36.	The FY2006/08 Project Document stated that PFTAC would improve its relevance by: (i) better articulating long term reform strategies, including specifying the linkages between programs and progress towards good practice standards; (ii) paying increased attention to highly focused capacity building activities that support the reform strategies of countries and other donors; and, (iii) concentrating more resources on longer term reform issues and continuing to provide ad hoc assistance to high priority areas. PFTAC was to play a more strategic role by helping the PICs to develop their own capacity to formulate and roll out their own long-term reform strategies and place greater emphasis in working with country management groups (such as steering committees or task forces) to articulate the detailed design of new systems. During the period under evaluation PFTAC worked more at the technical level than at the broad strategic level to define TA priorities. This reflects, in part, PFTAC’s limited resources and broad geographic coverage. During the evaluation period, there was little evidence of PFTAC undertaking broad, systematic stocktaking of activities to provide a base for regional efforts to find common solutions and to identify priority areas for the region, something that was suggested in the Project Document. However, recently PFTAC has undertaken more work to identify regional initiatives in the fiscal area. Overall, much of PFTAC’s assistance remains ad hoc in many countries. That being said, the Evaluation Team did receive feedback that one of PFTAC’s key strengths was to provide quick responses and advice that meets the needs of its clients. 

37.	An issue to consider is whether PFTAC is appropriately resourced, both in terms of the number of Resident Advisors and budget for short term experts, to effectively engage in the manner defined in the Project Document. All previous evaluations have concluded that resource constraints have limited the ability of PFTAC to fully respond to the demand for its services. This evaluation confirms these previous findings. Building capacity, as opposed to providing short term on demand advice, requires a deeper engagement over a period of years. This level of engagement cannot be met in a model in which each Resident Advisor is expected to cover most of the 15 countries, some of which have challenging institutional environments. In the words of one survey respondent “PFTAC has proven a useful short term advisory service and a help desk, one that operates timely and provides high quality and in-depth advice. Neither its mandate nor resources extend to long term capacity building nor does it focus on regional solutions.” PFTAC’s current model and resources are not consistent with the vision set out in the Project Document. Stakeholders need to consider whether they wish PFTAC to continue operating in the same way that it has since it was established 1993 or whether their vision is to increase the strategic relevance, effectiveness and sustainability of its assistance. Achieving this vision would require a significant increase in PFTAC’s resources. If the stakeholders do not provide the necessary resources, the objectives in the next Project Document should be scaled back to be more consistent with the available resource envelop and would involve providing more ad hoc than strategic advice. 

38.	While PFTAC should not lose its ability to respond quickly and flexibly to requests for assistance, a more strategic approach to planning its assistance should be considered. Key elements of such an approach would include: (i) considering the sum total of PFTAC’s assistance across the four functional areas at a country level; (ii) identifying strategic objectives that PFTAC hopes to achieve in a three year period in each country in each functional area; and (iii) identifying areas in countries that PFTAC does not expect to be involved, either because the PIC does not place priority on receiving such assistance or adequate assistance is being provided by other TA providers. In developing PFTAC’s work program, important strategic inputs to identify priority areas would continue to be made by APD and the three TA departments. Such an approach would have several advantages: (i) it would provide a framework in which to assess how individual PFTAC interventions are contributing to a strategic objective over a three year period; (ii) identify opportunities to develop synergies among the Resident Advisors; and (iii) provide a framework to help coordinate assistance provided by the donor community.       

C.	Effectiveness of PFTAC

1.	Overall Assessment of Effectiveness

39.	PFTAC’s assistance was effective in achieving results (see Table II.1). Survey respondents rated the quality of PFTAC’s work program, use of TA outputs and effectiveness of achieving results all Good (see Table C.14). These findings are broadly consistent with the conclusions reached by the Evaluation Team based on interviews and a review of documents. The positive feedback about the effectiveness of PFTAC in achieving results was consistent regardless of the employer of the respondent, the functional area in which government officials worked or the familiarity of the respondents with PFTAC’s work (see Tables C.15 to C.17). All modes of assistance were rated as Good in terms of effectiveness (see Table C.18). These findings echo the conclusions of the previous evaluations. As a result PFTAC was viewed as an excellent partner by the PICs and the first option in the Region for macroeconomic management advice.

40.	Survey respondents were asked to identify up to four factors that made an important contribution to PFTAC’s effectiveness. PFTAC’s location in the region and the high quality of its expertise received the most frequent mentions. During our fieldwork people consistently stated that IMF could not deliver similar services from Washington and that because PFTAC was in the region clients felt comfortable seeking ad hoc advice by telephone or E-mail. This was viewed as an important benefit by the officials interviewed by the Evaluation Team. The quality of PFTAC’s expertise was highly valued and was generally felt to be better than that of other organizations. Other important factors contributing to PFTAC’s effectiveness included its responsiveness to government priorities, consistent engagement over a period of years, personal relationships built up with its clients, knowledge of its client countries and good coordination with other TA providers and IMF Headquarters (see Table C.19). 

41.	Respondents were also asked to identify up to four areas where PFTAC could improve its effectiveness. The three most frequently cited areas included more follow up to help implement recommendations, more regional workshops/training courses and more attachments/secondments of Pacific Islanders. Other factors mentioned by more than 20% of the respondents included increasing the number of PFTAC staff, more use of short term experts, more support for regional initiatives, greater use of Pacific expertise and use of long term in-country advisors (see Table C.20). 

42.	One of the strengths of the PFTAC model is IMF’s backstopping, both from the TA departments and by the Resident Advisors for the short term experts. It contributed to the good quality assistance provided by the PFTAC and was generally appreciated by the Resident Advisors and senior government officials. Based on interviews with current and past Resident Advisors, the amount of back stopping varies across the three TA departments. A significant amount of the time of FAD staff is reportedly spent on backstopping, while MCM appears to be at the other end of the spectrum and its backstopping was limited. This partly reflects two factors: (i) fiscal assistance accounts for over half of all the assistance provided by PFTAC; and (ii) MCM went through three departmental re-organizations in the last few years which had the effect of markedly reducing MCM’s TA oversight structure. In terms of level of effort devoted to backstopping, STA appears to be between MCM and FAD. All three TA departments advised the Evaluation Team that the back stopping model was under more stress in 2009 than in 2005 because of the growing number of RTACs and the 2008 IMF budget and staff cutbacks. These factors are resulting in a mismatch between a growing workload and shrinking resources. IMF is addressing this challenge by implementing a more detailed time recording system that will allow donors to be charged for the actual backstopping costs and for the resulting funds to accrue to the budgets of the TA departments rather than to the central IMF budget. However, PFTAC will not be subject to actual costing until 2012. Any decision to expand PFTAC must be complimented by corresponding resources for backstopping.

43.	Survey respondents were asked to compare PFTAC assistance with that offered by IMF Headquarters and other TA providers. Relative to IMF Headquarters, nearly 90% of respondents agreed/strongly agreed that PFTAC responds more quickly, has a better understanding of the PICs and is more effective in supporting regional initiatives. Over 90% of the respondents felt that PFTAC supported the implementation of policies and strategies identified by IMF Headquarters. PFTAC’s expertise is viewed as being on par with that of IMF Headquarters with PFTAC’s Resident Advisors having the advantages of being located in the region, being accessible and well known to the country authorities and being very familiar with country needs and institutional constraints. The findings comparing PFTAC to other TA providers were similar (see Table C.22).

44.	The Evaluation Team identified a number of factors that contribute to TA effectiveness. Large majorities of the survey respondents agreed or strongly agreed that PFTAC assistance exhibited each of the following characteristics: (i) demand driven and responsive to country needs; (ii) strong country ownership; (iii) compliments IMF Headquarters TA and surveillance work; (iv) provides feedback to IMF Headquarters; and (v) PFTAC’s regional professional associations (e.g., AFSPC; PIFMA; PITAA) are useful for networking and learning (see Table C.23). Over the years, PFTAC has devoted a considerable amount of the time of the concerned Resident Advisors and has provided financial resources to develop and nurture AFSPC, PIFMA and PITAA. It was a sound strategic decision to support these regional professional organizations.   

45.	One way to assess effectiveness at the highest level is to try to determine where respondents believe that progress is being made in achieving the four objectives in the Project Document. The Evaluation Team expected that more respondents would have difficulty answering this question because PFTAC’s objectives are quite broad and are not readily monitorable or measurable. Most people replied that although substantial progress has been made, PFTAC’s objectives were not fully achieved. Between 30% and 40% reported that only modest or no significant progress had been made. While about a quarter of the respondents reported that the objectives related to financial sector supervision and AML/CFT had been achieved, less than one in five felt that the PFM, revenue administration and statistics objectives had been fully achieved (see Table C.24). One interpretation of these results is that there will be a continuing need for PFTAC assistance for the foreseeable future and that talk of an exit strategy for PFTAC is premature. Another possibility is that more work is needed in the next Project Document to refine and clarify the objectives so that they provide better benchmarks against which to monitor progress, or lack thereof. It is possible that more refined objectives would result in a more focused work program and possibly different strategies to achieve the desired objectives.

46.	PFTAC is filling an important niche role in the Pacific and all people interviewed by the Evaluation Team felt that the PFTAC brand name was excellent.  Because of its success, during the evaluation period there were calls to leverage PFTAC’s brand name by expanding it into areas that were outside IMF’s core competencies. However, it was decided not to follow that path. In the Evaluation Team’s opinion that was a sound decision. Key ingredients of PFTAC’s success are the international caliber expertise that it offers and IMF’s backstopping. Expanding PFTAC’s activities beyond IMF’s core areas of expertise would weaken what has been a successful model, since IMF would not be in a position to recruit, supervise or backstop people working outside IMF’s areas of competence. To ensure that its effectiveness is not undermined, any plans to expand PFTAC should focus on deepening its engagement in areas where IMF is acknowledged as a center of excellence rather than broadening its range of services into other areas which are beyond IMF’s core competencies. As one survey respondent said “PFTAC's success is based on focusing on key areas of IMF's expertise. This needs to continue in the future, at the expense of expanding into the areas that are not directly under the IMF's purview.” This does not mean that PFTAC should reject opportunities to co-locate with staff from other TA providers if they are working in related fields. Such co-location might result in better donor coordination, more synergies between donors and better sharing of knowledge, all factors that are consistent with the principles of the Accra Agenda for Action and should lead to a better achievement of development results. However, IMF would not be expected to recruit, supervise or backstop these people. That would be the role of the organizations financing them.

2.	Regional Workshops/Training Courses
 
47.	Regional workshops and training courses were mainly conducted in conjunction with ASFPC, PIFMA and PITAA meetings. Participants were asked to rate various aspects of the workshops on the PFTAC Evaluation Survey (see Table C.25). The topics covered, resource persons, quality of the presentations and quality of the venue were all rated on the border between Good and Excellent. The balance between theory and practice, time available to interact with other participants and length of the course were rated as Good, although about one fifth of the respondents felt that the courses were too short. The lowest rating, at the cutoff point between Modest and Good, was given to post workshop follow up and support. Nearly half of the respondents felt that improvements were needed in this area.

48.	Nearly all of the participants felt that examples from Pacific countries were particularly useful and less than 20% felt that the topics were too theoretical or too advanced for their organization. The effectiveness of workshops in achieving results depends largely on whether participants apply the knowledge and skills learned on the job. Most of the respondents either agreed or strongly agreed with statements that the training was relevant for their day to day activities (95%) and that they used the skills on the job (69%). Relatively few replied that they seldom used the skills on the job (18%), the topics were too advanced for their organization (16%) or that their organizations did not have the information technology or systems to apply the knowledge (7%) (see Table C.26). These findings indicate that the topics for the regional workshops were appropriate, that the training was well prepared and delivered and that the participants acquired knowledge and skills that they use on the job. The workshops would have a wider impact if all participants, as a matter of routine, shared the knowledge with other colleagues by conducting seminars when they returned to their offices. While some agencies require this, others do not. 

3.	Donor Coordination

49.	Donor coordination is a perennial issue among TA providers and was identified in the Project Document as an area needing improvement. Good donor coordination requires a willingness and ability of all parties to coordinate.  PFTAC undertook several initiatives to better share information with other TA providers: (i) posting its country notes on its website so all donor can learn the details of PFTAC’s past and future work programs in each country; (ii) sending reports to donors after missions; and (iii) meeting representatives of other TA providers and their consultants when on mission. While back to office reports the PFTAC sends to donors were useful, they only highlighted topics on which coordination was needed with specific donors. They did not highlight areas where donors could add value, notably in cases where PFTAC lacked the resources to address a given need. These reports could be improved by adding a section or a box covering the areas where synergies might be developed with other donors to support the implementation of recommendation. Overall, these initiatives have been successful in improving coordination with other donors. Less than 20% of the respondents identified donor coordination as a major area for improvement (see Table C.20).

50.	The Evaluation Team identified examples of PFTAC staff joining ADB missions to help in the diagnostics for policy based lending and the formulation large TAs to support the implementation of PFTAC recommendations in Vanuatu, Tuvalu, the Cook Islands and Kiribati. These are excellent examples of donors working together to achieve results. PFTAC’s contributions were to identify priorities, help scope out the assistance and help to peer review the work undertaken under the larger TA provided by ADB. There were also examples in the Solomon Islands of synergies between PFTAC and AUSAID’s PFM team and NZAID’s revenue administration team. Going forward every opportunity should be sought to use this model of donor coordination. 

51.	As part of the most recent PFTAC replenishment discussions it was agreed that two World Bank staff, one responsible for Public Expenditure and Financial Accountability (PEFA) and one for State Owned Enterprise (SOE) reform, would be co-located in the PFTAC office. While those staff are not formally part of PFTAC and IMF does not backstop their work, it was thought that locating World Bank staff in PFTAC would help to improve donor coordination and synergies by a greater sharing of information. Since the terms of reference for the Evaluation Team focus on the period from FY2006 to FY2008, formally evaluating this experiment was beyond the evaluation’s scope. There are potential synergies between PEFAs and the work of the PFM Advisor. PEFAs provide a detailed assessment of the fiscal sector in a country. Based on the PEFA analysis, a PFM action plan is prepared which provides a framework in which donors, including PFTAC, can prioritize and program their PFM assistance. Though less direct there are also potential synergies between the work of the PFM Advisor and the SOE specialist. Often PFM work identifies operating losses of SOEs as a major fiscal problem, something that is relatively common in the Pacific Region. 

52.	 As called for in the FY2006/08 Project Document, PFTAC now prepares a Country Strategy Note for each PIC that is updated periodically and is available on PFTAC’s website. For each country, separate matrices are prepared for each of the four functional areas that define the current situation including comments on organizational capacity, TA results and outcomes, advisor visits and the activities of other donors. While there is some scope for improvement to make them a more effective tool for reporting on the achievement of development results, the Evaluation Team found that these notes were useful in that they provided a concise summary of the situation, plans in the country and information that should facilitate donor coordination.

53.	While there were a few cases brought to the attention of the Evaluation Team where better coordination was desirable these appear to be relatively isolated instances and overall PFTAC is doing well in the area of donor coordination. Going forward, PFTAC will need to increase its coordination with the European Union. The European Union is providing budget support in the Pacific. There may be opportunities for PFTAC to help prepare and monitor the European Union’s budget support program. 

54.	Delivery of TA through the PFTAC is consistent with the calls in the 2003 Rome Declaration on Harmonization, the 2005 Paris Declaration on Aid Effectiveness and the 2008 Accra Agenda for Action for increased country ownership, greater donor harmonization and coordination and building institutional capacity (see Table A.5). The donors identified IMF as the international organization with the comparative advantage for macroeconomic management TAs and then provided funds to it. The PFTAC governance structure provides an explicit voice to the PICs and contributing TA providers in establishing priorities and monitoring implementation of the work program. By pooling their funds, donors improved coordination and reduced the transaction costs for beneficiary countries (e. g., fewer missions for TA providers addressing the same topic; less conflicting macroeconomic advice).

D.	Efficiency of PFTAC

1.	Overall Assessment of Efficiency

55.	PFTAC is a cost effective, lean organization (a Coordinator; four Resident Advisors and four support staff) with limited overheads. PFTAC generates a high volume of outputs in terms of missions and workshops for relatively modest resource inputs. PFTAC’s efficiency was rated as Good (see Table II.1). The PFTAC Advisors spend much of their time in the field, PFTAC responds quickly and flexibly to requests for assistance and is viewed as an organization with limited bureaucracy. All of these are characteristics of implementation efficiency. PFTAC’s efficiency was praised during field interviews and was clear when the Evaluation Team visited PFTAC’s offices and saw the white board that showed the mission schedules of the Resident Advisors. The perceptions of PFTAC’s process and implementation efficiency given by the survey respondents generally confirm the Evaluation Team’s conclusions. About one third of the respondents rated PFTAC’s efficiency as Excellent and nearly 60% as Good. Only about 10% rated PFTAC’s efficiency as Modest or Poor (see Table C.29). IMF staff and consultants ranked PFTAC’s implementation efficiency as Excellent and government officials and the staff of TA providers rated it as Good. The greater was the respondent’s familiarity with PFTAC the better implementation efficiency was rated (see Tables C.30 to C.32).

56.	In the past IMF’s time recording system did not link IMF staff time to specific TA activities. Thus the data available was not adequate to undertake a proper cost efficiency analysis to compare PFTAC with other RTACs and the cost of delivering TAs from Headquarters. IMF’s new time recording system should capture staff time spent on back stopping, TA delivery and administration. This should permit a better cost efficiency analysis of alternative IMF TA delivery modes in the future. Based on the available information, the Evaluation Team’s assessment is that PFTAC uses its resources cost efficiently. In terms of cost per person month of service delivered, PFTAC’s figures compare favorably to those of other RTACs and there are no significant differences across functional areas (see Table A.6). The cost of air travel in the Pacific is high but PFTAC tries to plan its missions to economize on costs. PFTAC has taken other measures to limit its operating costs (e.g., undertaking training as part of every mission; introducing the use of Skype to reduce communications costs). PFTAC support for AFSPC, PIFMA and PITAA is a cost effective form of outreach.  

57.	The FY2006/08 Project Document stated that PFTAC’s reporting would be strengthened by undertaking quarterly activity costing of each of its functions to breakdown total expenditures by: (i) the four Program Areas; (ii) the overall costs of outputs delivered to individual PICs under national programs and those delivered for region-wide programs; and, (iii) the three forms of TA delivered under each Program Area. Partly because of weaknesses in IMF’s financial management system for RTACs, PFTAC did not fulfill this commitment. Some improvements have been made in the financial systems over the years (e.g., OTM sends PFTAC monthly statements on draw downs against expenditures; PFTAC has access to the financial system). Although information is reported by PFTAC, the financial information is maintained at Headquarters, the PFTAC Coordinator does not have full and easy access to the data and more training is required on the use of available financial management tools. 

58.	One way in which cost efficiency could be further improved would be to make greater use of short term experts to leverage the fixed costs associated with the Resident Advisors. Relative to the previous funding cycle, there were increases in the FY2006/08 budget for professional attachments, short term experts, regional travel and seminar participants (see Table A.2). However, the ratio of PFTAC’s expenditures for short term experts to the cost of Resident Advisors was 33%. For the other RTACs, the corresponding ratio ranged from a low of 46% to a high of 76% (see Table A.6). This comparison suggests that there are opportunities to further leverage the expertise of the Resident Advisors, and thus improve cost efficiency, by making greater use of short term experts. While the PFM and Revenue Advisors are making good efforts in this area, there is room for the Financial Sector Supervision and Statistics Advisors to make more use of short term experts. This move would require the Resident Advisors to allocate more of their time to recruiting, supervising and backstopping the experts. The experience of the PFM and Revenue Advisors indicate that this can be done without significantly compromising the quality of PFTAC’s assistance, reducing the time the Advisors spend in the field or unnecessarily increasing PFTAC’s overheads and bureaucracy.

59.	In the longer term PFTAC may be able to use technology to further improve cost efficiency through the use of web based solutions for self learning and distance education. Doing so will depend on both the quality of the telecommunications infrastructure in the PICs and IMF’s investment in the necessary technology. However, a recent announcement that India, ADB and the University of the South Pacific will support ICT-based education or learning delivered through technology in the Pacific indicates that PFTAC should monitor developments in this area. The possible use of distance learning was raised in the Project Document for the current funding cycle but has not yet been seriously explored. 

2.	Evaluation and Monitoring

60.	The importance of PFTAC being able to better define and document its outputs and outcomes has been a constant topic at TRPC meetings since PFTAC was established.  This is also an important, ongoing issue for the other RTACs. In line with the recommendations of the 2004 Evaluation and statements in the FY2006/08 Project Document, the Evaluation Team confirmed that efforts were to be made to improve monitoring and evaluation by: (i) supplementing the Three-Month Rolling Work Plan with a table showing the objectives, strategies, functions and action timetables for each functional area in each PIC; (ii) submitting six-monthly reports to the TPRC on the performance status of ongoing activities by functional  area and country; (iii) making use of the Technical Assistance Information Management System (TAIMS) after it was redesigned to improve functionality and allow remote access; and (iv) undertaking one user satisfaction survey.  

61.	Consistent with the calls in the Paris Declaration and the Accra Agenda for Action, there was considerable emphasis in PFTAC Project Documents on developing a results-based monitoring system to track performance. A design and monitoring framework was included as an annex in the FY2009/11 Project Document and another annex summarized results achieved in the past and planned activities for each functional area in each country. Logical frameworks were to be used to monitor TA project implementation against specified objectives, outputs, and indicators using TAIMS. TAIMS was not a user friendly system and was not consistently used in IMF for most of the period covered by the evaluation. Information technology and communications infrastructure weaknesses compounded the problems for PFTAC and made it difficult to access the system. A review of TAIMS was undertaken in 2007 and recommendations were made for improvement. By mid-2008 an enhanced version of TAIMS was operational and as of 1 May 2008, IMF management required that all TAs, regardless of source of financing, be monitored using TAIMS. Indicators are to be developed for every TA to facilitate an objective assessment of TAs through the use of monitorable, verifiable indicators. Although it is too early to determine whether the enhanced version of TAIMS will result in IMF having a better TA management information system, many of the problems identified in the TAIMS usability assessment have been addressed.

62.	Despite the progress that has been made, PFTAC’s efforts to develop a good results based system remains an unfinished agenda. In interpreting this outcome, three factors must be recognized: (i) all TA providers are struggling to operationalize results based management systems; (ii) it is widely acknowledged that developing ways to measure and monitor capacity building TA is a major challenge; and (iii) as documented in the TA evaluation undertaken by IMF’s Independent Evaluation Office and the 2008 paper on improving the impact of TA, this is an IMF wide issue rather than just a PFTAC issue. As an institution, IMF is investing considerable time and resources to improve results based monitoring and reporting. PFTAC should benefit from this effort. However, measuring performance in a meaningful way and moving to a results focused management of TAs will take time and will be an evolutionary process.

63.	The Evaluation Team believes that, among other things, problems include the lack of clear, measurable objectives that PFTAC is trying achieve at the country level and the rather short time frame, one year, that IMF uses to plan and manage TAs. Typically, it takes several years to achieve capacity building results. If a one year time frame is used, it is likely that the monitoring and reporting will focus on inputs and short term achievements rather than medium to long term outcomes. PFTAC’s most successful TAs in terms of achieving results involved sustained input over two to three years. Missions built on, and extended, the accomplishments of earlier missions. It would be easier to report on results if a medium term strategic objective of what was to be accomplished in about three years was set to provide a framework to access the progress made by each mission or in a particular year. This approach would provide a framework to assess the many small steps that are needed to achieve results in the area of capacity building, while keeping in view the big picture and strategic objective. That should make it easier to report on the outcomes achieved or that are on track to be achieved and to take corrective action if there appears to be a problem.  

E.	Sustainability of PFTAC

1.	Overall Assessment of Sustainability

64.	Evaluating sustainability is a challenge because many TAs are works in progress and their sustainability will only be known in the future. While the Evaluation Team found examples in all the functional areas of PFTAC assistance resulting in changes that were embedded in the policies and procedures of the beneficiary agencies, there were other examples of sustainability being undermined by organizational weaknesses and staff turnover or a lack of follow up and support to implement the recommendations. Over half of the survey respondents rated the sustainability of TA benefits as Good and about 20% rated it as either Excellent or Modest (see Table C.33). Respondents working for government agencies and IMF staff/experts, on average, assigned a Good rating to sustainability. On average, staff of TA providers gave a Modest rating for sustainability – 44% gave a Modest rating and 11% gave a Poor rating (see Table C.34). There were some differences in the views of government officials on sustainability depending on where they worked. Although a Good rating was given by officials from all agencies, officials from central banks and revenue administration were more likely to give an Excellent rating and officials from statistical agencies were more likely to assign a Modest rating (see Table C.35). People who were very familiar with PFTAC were more likely to assign higher ratings to sustainability than were people who had limited interaction with PFTAC (see Table C.36). 

65.	Institutional weaknesses undermine the sustainability of PFTAC’s assistance. Three quarters or more of the survey respondents agreed or strongly agreed that the following factors adversely affected sustainability: (i) staff turnover and loss of trained staff; (ii) staff shortages; and (iii) budget shortages. Although nearly two thirds of the respondents felt that political changes sometimes undermined sustainability about a quarter disagreed/strongly disagreed with this notion (see Table C.37). As one survey respondent stated “While substantial progress has been made in many areas with PFTAC assistance, the narrow staffing and skills base in the beneficiary countries means that those reforms are under continual threat from losses of staff through natural wastage, immigration and transfers to regional organizations.” Sometimes there is a lower than desirable take up on the recommendations because the mechanisms in some countries for obtaining and then implementing commitments are weak. Sometimes there is no local counterpart assigned to follow up on the recommendations when the Resident Advisor hands over his/her report. In such cases there is a tendency for little to be done until the next visit of the Resident Advisor.  Largely because of these factors, the Evaluation Team rated the sustainability of the benefits of PFTAC’s assistance as Modest (see Table II.1).

66.	 Although the recommendations of many TAs were implemented, survey results and the Evaluation Team’s field work indicate that for some TAs this was not the case. In some cases government officials agreed with, and wished to implement the recommendations, but the necessary resources were not available. All PICs face budget constraints and there are many competing priorities for the limited funds available. For TAs to be successful in achieving the desired outcomes, executing agencies sometimes need more than the advice provided by PFTAC. Sometimes implementing recommendations requires funding. Such cases often involve support from a TA provider to finance long term advisors to help with the detailed work necessary to implement PFTAC’s recommendations or to finance investments in information technology software and hardware or to undertake surveys to strengthen statistics. Neither PFTAC nor IMF provide such financing.  The need for more follow up and support for the implementation of recommendations was identified as an area for improvement by survey respondents (see Tables C.20 and C.25). While PIC authorities must do their part to ensure that there are sufficient budgets and numbers of well qualified staff to implement the recommendations emanating from PFTAC’s assistance, the Evaluation Team believes that in situations where the success and sustainability depend on additional financial support, PFTAC must find ways to help the beneficiary agencies secure the necessary funding. The first step is to clearly define and cost the resources needed to implement the resulting recommendations. This should be mandatory for all TAs. Once the costs are known, a strategy can be developed to secure the necessary financing. Although the Evaluation Team identified some instances where this happened with ADB, AUSAID or NZAID, this approach needs to be used more frequently in the future.

2.	Use of Pacific Expertise

67.	The FY2006/08 Project Document states that where the relevant skills are available, PFTAC will use experts from within the region and that national authorities would be asked to nominate officials who could undertake short term assignments. The Project Document also states that, as recommended in the 2004 Evaluation, PFTAC would introduce a new tier of locally recruited consultants, who were expected to be mature professionals with research and policy advisory experience, to support the Resident Advisors. Depending on the needs they could be seconded to PFTAC for a period of three to six months and would then return to their agencies. Feedback from Resident Advisors indicated that this proposal was not accorded a high priority by the TPRC because the PICs were concerned about the risk of losing some of their small pool of trained staff.

68.	PFTAC has made limited use of Pacific expertise. A Pacific Islander has never been employed as a Resident Advisor. The Evaluation Team identified only a small number of instances when Pacific expertise was used by PFTAC. Examples include: 
(i)	A staff who compiled BOP statistics in the Reserve Bank of Vanuatu conducted two missions to the Solomon Islands. The authorities reported that they benefited from these services. She was technically qualified and because she had addressed similar issues in her home country and her knowledge of Pacific Island culture and traditions, she could propose improvements that were feasible in the Solomon Islands’ context. However, because the revised BOP estimates and the recommendations remain unpublished, the assistance cannot be deemed to have been fully effective in achieving the desired results. The Central Bank of the Solomon Islands would like to continue this relationship by seconding one of their staff to the Central Bank of Vanuatu, something that it viewed as necessary for the assistance to achieve the desired results in a sustainable manner.  
(ii)	An internal auditor from Fiji was recruited to assess internal audit in the Pacific Region based on country presentations at a PIFMA workshop and visits to the Cook Islands, Fiji, Palau and Tonga. The report found that there were significant weaknesses in the internal audit process in the region and that weaknesses in financial accountability, the controlled environment and the internal control framework undermined good governance and led to fraud, corruption and misappropriation of funds. The report included a number of recommendations on how to strengthen internal audit. Overall the Fiji expert performed well. 
(iii)	 One noteworthy example involved attaching a staff member of the Reserve Bank of Fiji to the National Reserve Bank of Tonga as the Acting Deputy Governor for Corporate Restructuring. This person provided useful support.
(iv)	A staff from the PNG Central Bank was attached to the Cook Islands Financial Supervisory Commission to help strengthen supervisory practices.
(v)	 An official from PNG gave lectures in Tonga on PFM covering gender budgeting, bench marking PFM progress and strategic planning.

69.	The majority of survey respondents, about 80%, felt that it was either highly important or important for PFTAC to promote the use of Pacific expertise (see Table C.38). However, PFTAC’s efforts to do so were rated as Modest, the lowest rating given on the survey for any criteria (see Table C.39). PFTAC should make more of a concerted effort to identify, develop and use Pacific expertise. That being said, the emphasis should be on expertise rather than nationality of the expert. This point was made by several people interviewed by the Evaluation Team. It is essential that PFTAC maintains its high standards of only providing experts who are truly well qualified and experienced. PFTAC’s reputation will suffer and its clients will be ill served if they receive poor advice from people who are recruited primarily on the basis of their nationality rather than their experience and expertise. 

70.	PFTAC’s limited use of regional expertise in contrast to other RTACs. In CARTAC and the AFRITACs people from the region were employed as center coordinators, resident advisors or as short term experts. However, there is clearly some expertise in the region, either officials currently working for government agencies, ex-officials, retired IMF, World Bank or ADB staff or Pacific Islanders living overseas, particularly in Australia, New Zealand and the United States, who may be interested in contributing to the development of the region by working for PFTAC.

71.	Given the feedback in this evaluation for the increased use of regional experts, PFTAC should revisit the 2004 proposal and develop alternatives. If there is still reluctance at the TPRC level to adopt the 2004 proposal, a scheme targeting young professionals from the region for 12 month attachments to the Resident Advisors could be considered. The young professionals could be bonded to work with their agencies for a period of time after completing the attachment. The young professionals would require considerable mentoring time from the Resident Advisors. Given that the Resident Advisors are over stretched, the young professional scheme could be considered for introduction only if the number of Resident Advisors increases.

F.	Tripartite Review Committee
72.	Periodically there have been discussions about the need for more frequent TPRC meetings. The decision was made to maintain the schedule of meetings about every 18 months but to supplement it by issuing Six Monthly Update reports and improving the PFTAC web page. Initially the Evaluation Team was skeptical that one TRPC meeting every 18 months was adequate. However, most people interviewed felt that the TPRC was functioning well, that the meeting frequency was appropriate given that progress reports are now circulated every six months, the informal meetings between the TPRC chair and the PFTAC Coordinator and the additional information available on PFTAC’s web page. The TPRC meetings are not dominated by IMF staff and donor representatives. Representatives of the PICs actively participate and express their views. Key informants noted that the TPRC members are senior people who have busy schedules and that time and costs associated with TPRC meetings are not insignificant. About 70% of the survey respondents felt that the frequency of meetings was appropriate while nearly one third felt that more frequent meetings were desirable (see Table C.27). 

73.	One cost-effective way to increase the effectiveness of the TPRC would be for it to meet before or after the annual Forum Economic Ministers (FEM) meeting. PFTAC presentations to FEM have been appreciated and the PICs look to IMF for macroeconomic advice and forecasts, particularly during times of global crises. The FEM discusses broad policy issues, some of which are related to PFTAC’s areas of expertise. However, sometimes more focused discussions are required to translate FEM’s decisions into action. Several senior officials interviewed by the Evaluation Team indicated that synchronizing the timing of the meetings would be a cost effective way to strengthen the link between the regional policy initiatives adopted by FEM and the work of PFTAC.  

74.	Most survey respondents rated the TPRC as Good in contributing to PFTAC’s effectiveness by providing oversight and guidance, promoting country ownership and facilitating donor coordination (see Table C.28). About 30% of the respondents assigned Modest or Poor ratings to providing oversight and guidance and promoting country ownership. The Evaluation Team received some feedback that the TPRC could be streamlined, focus more on strategic policy issues and more actively monitor PFTAC’s work plan Steps could be taken to enhance the accountability of PFTAC to the TPRC by adopting a more structured reporting format based on the PFTAC work plan and related milestones that place more emphasis on the achievement of outputs and outcomes rather than on inputs, the implementation of TA recommendations and identifying regional priorities that could be pursued in several countries simultaneously. However, it is important to ensure that the TPRC continues to provide broad strategic advice and monitors PFTAC’s performance and does not stray into areas that are better left to the management of PFTAC.   

G.	Lessons for the Future
1.	Improving Organization and Management

75.	Some of IMF’s policies and procedures governing RTACs impact on PFTAC’s efficiency and effectiveness. Because many of the findings in this section are similar to those in the recent evaluation of the AFRITACs, in which one member of the Evaluation Team participated, the level of explanation is kept to a minimum in this report. Many of these issues were discussed and broadly endorsed at a retreat of RTAC Coordinators in December 2008. What remains to be done is for IMF to take action to address these issues. A new, expanded handbook is needed to codify the operational, management and administrative procedures for RTACs. The evaluation has identified several issues that need to be considered in the process of preparing the manual:

(i)	Empower the PFTAC Coordinator: The PFTAC Coordinator is an experienced professional. His work with both a functional and a regional department has given him broad exposure to the IMF’s activities and operations. However, IMF is not making full use of his technical expertise in PFTAC’s management and operations. The PFTAC Coordinator is not formally involved in the selection or supervision of the Resident Advisors. The PFTAC Coordinator should have a formal role in the selection and supervision of Resident Advisors and be able to initiate the recruitment process about nine months before a vacancy is likely to occur. The technical experts in TA departments should continue to identify suitable candidates from their rosters, assess technical skills of applicants and backstop the technical aspects of their work. 
(ii)	Lengthen the Appointment of Resident Advisors: Resident Advisors are appointed for one year, and can be renewed for one or more years. Capacity building is a medium to long term objective. It is not possible for Resident Advisors to make a significant impact in the area of capacity building in one year. The Evaluation Team received feedback from some officials that it was only after three years that they were fully conversant with all of their countries and were starting to have maximum impact. Because moving to a different country to take up a new assignment has implications for families and careers, a one year contract may discourage some well qualified candidates from applying. The Evaluation Team believes that Resident Advisors should initially be recruited for three years, including a one year probationary period. They could then be extended, depending on their performance and the demand for their services, for one and in some cases two years. The Resident Advisors should be subject to an annual performance review. The review would deal with performance issues, including possible non-renewal of probationary appointments or subsequently if performance was found wanting, if the demand to his/her expertise diminishes or if funding is not available. The Evaluation Team believes that that a well managed annual performance assessment system for Resident Advisors would be sufficient to offset risks that some may believe are associated with moving away from the current system of one year renewable appointments. The Coordinator and all Resident Advisors should also be formally briefed by their predecessors and attend orientation sessions in Washington before assuming the job. 
(iii)	Improve the Recruitment Process for Resident Advisors: The transparency of the recruitment process for Resident Advisors needs to be improved. All such positions should be advertised. While some RTAC positions have been advertised, some TA Departments have resisted external advertisement and limited the pool of candidates to people on their rosters. Advertising would not prevent the consideration of candidates on the rosters. 
(iv)	Remove Barriers in IMF Policies and Procedures that Discourage Staff in TA Departments Working as Resident Advisors: In 2005 it was thought that efficiency gains could be made in reducing the involvement of Headquarters staff in backstopping by assigning staff from the TA departments to RTACs. Footnote 9 in the January 2006 RTAC Operational Guidance Note states that Resident Advisors may be IMF staff. However, some IMF policies and procedures discourage IMF staff from considering a position as a Resident Advisor. To become a Resident Advisor, IMF staff must go on an unpaid leave of absence, so pension benefits would not accrue during their tenure in PFTAC. There are also issues related to the benefit package and education allowances. The Evaluation Team believes that many staff of the TA Departments would benefit from spending time in PFTAC or an RTAC more generally at some point during their careers. However, not all PFTAC Advisors should be IMF staff. There should be a balance between IMF staff and new people with significant regional expertise.
(v)	Strengthen Financial Management and Control: The PFTAC Coordinator does not have all of the necessary tools to maintain budgetary control. To reinforce budgeting, controls, accountability and reporting, simplified budget monitoring and expense reporting tools should be introduced that would give the PFTAC Coordinator full and easy access to real time information with the ability to analyze it.
(vi)	Promote Learning Between PFTAC and CARTAC: PFTAC and CARTAC deliver broadly similar products to large groups of island nations. A more formalized system of information exchanges between the Center Coordinators and Resident Advisors would help the two RTACs learn from one another. Center Coordinators and Resident Advisors would particularly benefit from learning from their peers during the first two years of their assignments.
(vii)	Improve RTAC WebPages: WebPages can be useful tools for information dissemination, donor coordination and outreach. IMF’s webpage lacks a section on RTACs and also has no links to the WebPages of PFTAC or any other RTAC. Given the importance of the RTACs to IMF’s TA program, IMF should add a section and related links on RTACs to its webpage. In the longer term OTM, the RTACs and the Office of External Relations should consider whether a common portal for RTACs should be hosted on the IMF server. There would be a general page maintained by IMF’s External Relations Department where some of the high-level RTAC documents would be placed and links to the RTAC specific web pages. The RTACs would be responsible for maintaining the content related to them. The use of the IMF server would ensure that a high capacity, well maintained server is available and that the portal was well designed and professionally managed. It would also avoid the need for the RTACs to try to secure professional website programming and management expertise in markets where the availability of such skills are limited and reduce issues related to servers and connectivity. While PFTAC’s webpage is good there is room for improvement. It contains information about PFTAC in general, its areas of expertise and its staff, a small e-library and contact information for its staff, as well as for staff in both the IMF and PFTAC member countries. Its report section includes PFTAC Project Documents, six-monthly reports on activities and its rolling six-month work program. There are also links to ASFPC, PITAA and PIFMA websites. Each association’s webpage contains information on its goals and activities, and links to relevant publications such as PFTAC handbooks, IMF working papers and other documents of interest. The AFSPC webpage also includes a summary of regulatory regimes in the region and Financial Soundness Indicators for selected countries. On the negative side, PFTAC’s webpage lacks a site map and a search engine. Some of the hyper-links to the associations’ WebPages and a number of the links to documents did not work when tested by the Evaluation Team. Also, the PFTAC handbooks could not be found in a single convenient location. Information on TPRC meetings might also be useful to include on the website. 
(viii)	Extend the RTAC Financing Cycle to Five Years: Since it was established PFTAC has been on a three year funding cycle. There are considerable costs for both donors and IMF associated with replenishments. Papers must be prepared, negotiations must take place, evaluations must be undertaken and reports prepared to authorize the commitment of the funds. It would be more efficient to spread replenishment overhead costs over five rather than three years. Extending the financing cycle to five years would also be consistent with the call in the Accra Agenda for Action to increase medium-term aid predictability. 

2.	Control of Corruption in PFTAC Client Countries

76.	During the past decade, supporting efforts to reduce opportunities for corruption has become part of the development agenda. ADB, AUSAID and NZAID Pacific Strategies and the Pacific Plan identify the importance of strengthening anti-corruption institutions. The World Bank’s control of corruption index shows a mixed performance in this area across the PICs (see Table A.7), although most PICs are performing better with regard to controlling corruption than is the norm in other developing regions although corruption is problem in the Solomon Islands, despite some improvements since 2000, and in PNG and Tonga. 

77.	IMF’s approach, to combating corruption emphasizes prevention, concentrating on measures to strengthen governance and limiting the scope for corruption in two areas: “(i) improving the management of public resources through reforms covering public sector institutions (e.g., the treasury, central bank, public enterprises, civil service, and the official statistics function), including administrative procedures (e.g., expenditure control, budget management, and revenue collection); and (ii) supporting the development and maintenance of a transparent and stable economic and regulatory environment conducive to efficient private sector activities (e.g., price systems, exchange and trade regimes, and banking systems and their related regulations)”. IMF promotes sound oversight and operation of the internal control, auditing, and public financial reporting mechanisms and helps to improve accountability by enhancing transparency in line with internationally recognized standards and codes and combating money laundering and the financing of terrorism.

78.	Some PFTAC assistance promoted transparency and accountability and were consistent with the types of initiatives mentioned in IMF’s corporate documents that address anti-corruption measures. Examples of PFTAC assistance that resulted in practical measures that should help to reduce opportunities for corruption include:    
(i)	introducing systems and procedures to improve transparency  and accountability;
(ii)	Anti Money Laundering/Combating Financing of Terrorism activities; 
(iii)	strengthening public expenditure controls and expenditure tracking;
(iv)	supporting better cash-management to improve transparency, accountability, control and auditing; promoting sound accounting systems;
(v)	supporting internal audit, including reviewing the status of internal audit in the region and proposing a way forward;
(vi)	making tax procedures simpler, less complex and more transparent;
(vii)	reducing tax avoidance; 
(viii)	strengthening the regulatory frameworks and their application for financial sector supervision; and,
(ix)	strengthening bank licensing procedures.

3.	Other Lessons for the Future

79.	The evaluation identified a number of other general lessons for the future. Addressing the first three lessons would require endorsement of TPRC and the last four would need to be addressed by IMF:
(i)	Addressing PFTAC’s Limited Resources: PFTAC’s small scale has limited its ability to play an optimal role in supporting macroeconomic capacity building in the region. Consideration needs to be given to ways to scale up PFTAC.
(ii)	The TPRC Is Effective: Although the TPRC functioned well, there are areas for improvement: (a) synchronizing TPRC and FEM meeting; (b) enhancing the accountability of PFTAC to the TPRC by adopting a more structured reporting format based on the PFTAC work plan and related milestones that place more emphasis on the achievement of outputs and outcomes rather than on inputs, the implementation of TA recommendations and identifying priorities that could be pursued in several countries simultaneously.
(iii)	More Use of Regional Expertise: PFTAC needs to explore innovative mechanisms to increase its use of regional expertise. 
(iv)	Backstopping Contributes to PFTAC’s Success: Although backstopping is one of the strengths of the PFTAC model, it is under more stress because of the growing number of RTACs and the 2008 budget and staff cutbacks. IMF is addressing this challenge by implementing a system that will allow donors to be charged for the actual backstopping costs. Any decision to expand PFTAC must be complimented by resources for backstopping.
(v)	Providing More Information to Donors: The back to office reports to donors should highlight areas where donors could add value by adding a section or a box identifying areas where synergies could be developed to support the implementation of recommendations. 
(vi)	Monitoring and Evaluation: Despite the progress that has been made, PFTAC’s efforts to develop a good results based system remains an unfinished agenda. As an institution, IMF is investing considerable time and resources to improve results based monitoring and reporting. PFTAC should benefit from this effort.
(vii)	IMF RTAC Policies and Procedures: When OTM prepares an RTAC manual it should cover: (a) ways to empower the Resident Coordinators; (ii) lengthening the appointment term for Resident Advisors; (c) improving the process for recruiting Resident Advisors; (d) removing barriers in IMF policies and procedures that discourage staff in TA departments working as Resident Advisors; (e) strengthening financial management and control; (f) promoting learning between PFTAC and CARTAC; (g) improving RTAC WebPages; and, (i) extend RTAC financing cycle to five years.

III. ASSESSMENT OF FISCAL ASSISTANCE

Key Messages

•	The performance of PFTAC’s Public Financial Management (PFM) and the Revenue Administration assistance were both rated as Good.  Relevance, effectiveness and efficiency were all rated as Good but concerns about sustainability resulted in a Modest rating for that dimension of evaluation. 
•	PFM and Revenue Administration both received Excellent ratings for consistency with Headquarters activities and process and implementation efficiency. PFM’s efficient use of resource was also rated as Excellent.
•	With its strong IMF branding, PFTAC is viewed as the premiere fiscal assistance provider in the Region with fast response times, flexible and practical adaptations to local conditions and effective promotion of regional initiatives.
•	The increasing demand for fiscal assistance was accommodated by the active use of short term experts and increasing collaboration with other TA providers. Coping with the increasing range of demand, particularly in the PFM area, is a major challenge.  
•	More formal collaboration among TA providers is needed to develop common fiscal diagnostic and monitoring frameworks. 
•	Quality assurance provided by FAD’s backstopping was effective and efficient. Resource constraints could undermine this in the future. 
•	PFTAC has progressively aligned its fiscal activities with the Pacific Plan and FEM resolutions.
•	PITAA and PIFMA have been successful and their meetings are planned around the FEM meetings to improve cost effectiveness and level of representation. The challenge is to make these regional organizations financially sustainable in the long run.
•	Capacity constraints in the PICs is a potential threat to sustainability and will continue to generate strong demand for the fiscal assistance services for the foreseeable  future	

A.	Introduction to the Fiscal Assistance

80.	In the fiscal area, the FY2006/08 Project Document states that PFTAC will provide:
(i)	Public Financial Management (PFM) assistance: At the national level, PFTAC will: (a) support the formulation and/or review of implementation of PFM strategies and action plans; and (b) provide assistance to strengthen budget planning, preparation, execution and reporting, government accounting and debt management systems and internal control and audit capacity. Planned regional PFM activities included: (a) identifying fiscal reform initiatives that would support regional integration and free trade; (b) promoting good governance and accountability; (c) supporting sound debt management as a component of improved fiscal management; (d) encourage regional practitioner interaction by establishing PIFMA, a practitioner's forum; and (e) developing training materials to be made available on PFTAC’s web site; and(f) supporting a PFM course in partnership with the University of the South Pacific. 
(ii)	Revenue Administration assistance: At the national level PFTAC will: (a) help diagnose revenue administrations and develop strategic reform and modernization plans; (b) advise on the administrative and economic efficiency of existing indirect taxes, corporate and personal taxes, payroll and social security contributions, property taxes and tariff policies; (c) help improve the effectiveness of tax and customs administration including increased tax compliance, strengthened collection, audit, and taxpayer services and establishing special collection arrangements for larger business taxpayers; and, (d) support that implementation of new taxes and introduction of measures to bring medium and small taxpayers into the tax net. Regional activities were to include: (a) working with FAD to undertake a regional review of areas where fiscal revenue may be enhanced; (b) supporting PITAA; (c) exploring with FAD, donors and other organizations, the desirability of a regional customs reform framework; (d) undertake regional training seminars for customs and taxation administrators; and (e) preparing standard operational manuals, guidelines and publicity and education material for revenue agencies.

81.	During the evaluation period PFM activities mainly focused on budget formulation processes and presentation, revenue and expenditure forecasting and execution, cash and commitment management, general monitoring and reporting. Work on cash and debt management also took place. Revenue forecasting work was carried for the Marshall Islands, Palau and Tonga. In the cash and commitment management area work was undertaken in Kiribati, the Marshall Islands, Nauru and Tuvalu. More advanced budgeting work took place in Fiji and Samoa to support performance budgeting and medium term fiscal frameworks. Work on financial instructions took place in Palau and the Solomon Islands and debt management work was undertaken in Fiji. As planned, PIFMA was established and the country interventions were supplemented by annual PIFMA workshops on medium term fiscal frameworks, cash and commitment management and debt management.

82.	 There were 32 missions undertaken by the PFM Advisors, about one per month, 101 people attended workshops, there were 21 missions by short term experts and 10 secondments (see Table D.1). The PFM Advisors were spread thinly and averaged one or more missions per year to only six of the 15 PICs. The PFM Advisors did not have major activities in several countries during the three year period (FSM; Niue; PNG; Tokelau; Tuvalu). However the PFM Advisors were able to keep in touch with all the 15 PICs at the PIFMA meetings and workshops. The Cook Islands, Fiji and Palau each accounted for about 10% of the workshop attendees and four countries accounted for about half of the short term expert missions.  

83.	For Revenue Administration, the focus was mainly on the diagnostics, design and implementation of revenue policy and administration reforms for FSM, Niue, Palau, PNG, Solomon Islands, Tonga, Tuvalu and Vanuatu. The Revenue Advisor worked closely with the Legal Department in supporting legislation for tax reforms for FSM, Kiribati, PNG, Samoa, Solomon Islands and Tonga. The Legal Department provided both direct missions and technical backstopping of PFTAC’s short term experts. In PNG a 2007 Legal Department mission prepared the Internal Revenue Commission Administration Bill. In Tonga, a short term expert was engaged in 2005 to draft the Customs Act and associated regulations. There were also cases in which the draft legislation prepared during the evaluation period has not yet been enacted (e.g., the draft Value Added Tax and modernized tax legislation in the Solomon Islands; legislation for tax self assessment and a presumptive tax in Samoa). In the customs area, training was provided to FSM and Palau for post clearance audits. The regional workshops were mainly for practical training based on PITAA operational manuals supplemented by discussions on country experience in the Region. OCO provided presentations on customs administration. 

84.	There were 29 missions undertaken by the Revenue Advisor, 84 people attended revenue workshops, there were 22 missions by short term experts and 7 secondments (see Table D.2). The Revenue Advisor was thinly spread and averaged one or more missions per year over the three year period to only five countries and eight countries were visited only once in three years. Tokelau was not visited. The limited coverage of PNG was offset to some extent by the fact that PNG and FSM together accounted for about half of the short term expert missions and nearly a quarter of the people attending revenue administration workshops were from PNG.    

85.	Because many PICs have limited capacity to identify their priority needs, the Resident Advisors adopted general scoping approaches to identify country needs for fiscal assistance at the country level. The 15 PICs were categorized based on the level of their fiscal systems development, relative size, commitment to reform, activities of other TA providers and potential of the country having a demonstration effect in the Region. To prioritize country needs PFTAC used feedback from its missions, information from Article IV consultations, regional/country strategy notes, the minutes of the TPRC meetings, meetings of the regional groups like PIFMA, PITAA, OCO and FEM, the Pacific Plan and information from TA providers like AUSAID, NZAID, ADB and World Bank. The work plans were country driven and were kept flexible. 

86.	In the PFM area, the relatively light workload on macroeconomic and fiscal policy, compliance and external scrutiny and accountability identified as focal areas in the FY2006/08 Project Document reflected the relatively higher priority that the PICs accorded to basic budget formulation, execution, cash and commitment management and reporting systems. In the future the PFM focus should be enhanced by the introduction of PEFA as a diagnostic tool to evaluate public expenditure and financial accountability systems. It remains to be seen what level of resources the World Bank will devote to support the preparation of PEFAs in PICs. 

87.	For Revenue Administration, PFTAC’s lack of emphasis on customs reflected the preference of the PICs to use OCO to support customs administration functions. Also, the customs area was relatively more advanced than revenue institutions. Because of these factors it was appropriate for PFTAC to widen and deepen its support for tax administration rather than trying to cover customs with its limited resources. The production of handbooks and operations manuals was made possible by this focused approach on tax administration.

88.	The focus of regional activities centered on the increasingly closer links being forged among the regional associations like PIFMA, PITAA and OCO and with the Pacific Islands Forum Secretariat (PIFS) and FEM meetings. The role of the PIFS in promoting the Pacific Plan to coordinate common development strategies at the regional level enhanced the complementarity of the PFM and Revenue Administration work plans and regional initiatives. Closer links were also developed between PFTAC and the PIFS in monitoring the FEM principles of accountability. 

B.	Rating the Fiscal Assistance

89.	The performance of both PFM and Revenue Administration assistance, was rated as Good (see Table III.1). Relevance, effectiveness and efficiency were all rated as Good with efficiency bordering on Excellent. Concerns about sustainability resulted in a Modest rating of that criteria. With its strong IMF branding, PFTAC is viewed as the premiere fiscal assistance provider in the Region. In the words of one survey respondent “Overall impression is that from a capacity/authority perspective PFTAC advice in the region is seen as being synonymous with IMF Headquarters advice, albeit at times with a more practical approach more closely aligned with country capacity and priorities. In that regard, reflecting the limited capacity of Washington based staff to interact expeditiously with small countries in the region, at least in the area of public financial management, PFTAC is seen as the first port of call.” PFTAC’s 2007 member survey reported that most countries (3 out of 6) mildly agreed with the statement that PFM assistance was effective while 4 of 7 countries strongly agreed that the Revenue Administration assistance was effective.  

Table III.1: Rating PFTAC’s Fiscal Assistance
(Note: This information replaces a table that is available in the PDF version of this report)
Input of Resident Advisors (Person Months)
Weight: N/A per cent; Public Financial Management: 36; Revenue/Customs Administration: 32; Total: 68
Relevance
Weight: 20 per cent; Public Financial Management: 3.0; Revenue/Customs Administration: 3.0; Total: 3.0
Effectiveness
Weight: 40 per cent; Public Financial Management: 3.2; Revenue/Customs Administration: 3.2; Total: 3.2
Efficiency
Weight: 20 per cent; Public Financial Management: 3.6; Revenue/Customs Administration: 3.2; Total: 3.4
Sustainability
Weight: 20 per cent; Public Financial Management: 2.4; Revenue/Customs Administration: 2.4; Total: 2.4
Total rating
Weight: 100 per cent; Public Financial Management: 3.1; Revenue/Customs Administration: 3.0; Total: 3.0
Note: Column weights were defined by the Evaluation Team and row weights are based on the proportion of person months of Resident Advisor used to deliver each group of TA activities.
4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor
Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

C.	Relevance of the Fiscal Assistance

90.	The relevancy of both the PFM and Revenue Administration assistance were both rated as Good (see Table III.2). They were consistent with government priorities, PFTAC played a role in helping countries to identify their TA priorities and PFTAC’s engagement and formulation of fiscal assistance was Good. The broad relevance of work in the fiscal area is clear from the fiscal issues highlighted in IMF’s book on Pacific Island Economies, which included: (i) the efficiency of the tax system needed to be improved by improving tax administration, broadening the tax base and reducing tax incentives; (ii) measures needed to be instituted to limit the spill over to the fiscal accounts of highly variable nontax receipts; (iii) spending on public sector wages and salaries must be redirected to education, health and infrastructure; and (iv) budget monitoring techniques must be improved to more closely track spending. In the PFM area needed improvements included: (i) improving the quality and efficiency of public spending; (ii) more closely linking annual budgeting with medium term development strategies; (iii) improving expenditure control and accountability mechanisms; (iv) strengthening internal control mechanisms; and (v) limit subsidies to public enterprises. The PICs need to strengthen their revenue efforts and efficient revenue mobilization by: (i) simplifying the tax systems, particularly income tax; (ii) improving tax compliance and collection; (iii) expand the tax base by reducing exemptions; (iv) improving the administration of the Value Added Tax; and (v) improving customs administration to facilitate trade and increase collections. PFTAC’s fiscal assistance accords well with these issues.

Table III.2: Relevance of PFTAC’s Fiscal Assistance
(Note: This information replaces a table that is available in the PDF version of this report)
Input of Resident Advisors (Person Months)
Weight1: N/A per cent; Ratings2—Public Financial Management: 36; Revenue/Customs Administration: 32; Weighted Total: 68
Consistency with government priorities
Weight1: 50 per cent; Ratings2—Public Financial Management: 3.0; Revenue/Customs Administration: 3.0; Weighted Total: 3.0
Defining priorities
Weight1: 10 per cent; Ratings2—Public Financial Management: 3.0; Revenue/Customs Administration: 3.0; Weighted Total: 3.0
Quality of TA formulation and engagement
Weight1: 40 per cent; Ratings2—Public Financial Management: 3.0; Revenue/Customs Administration: 3.0; Weighted Total: 3.0
Total rating
Weight1: 100 per cent; Ratings2—Public Financial Management: 3.0; Revenue/Customs Administration: 3.0; Weighted Total: 3.0
1 Column weights were defined by the Evaluation Team and row weights are based on the proportion of person months of Resident Advisor used to deliver each group of TA activities.
2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor
Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

91.	About one third of the survey respondents assigned an Excellent rating to the consistency between the fiscal assistance and government priorities (see Table C.11). Over 90% of the respondents who were directly involved with PFM and Revenue assistance rated consistency with government priorities as either Excellent or Good (see Tables D.4 and D.5). This result confirmed feedback received from senior officials in the ministries of finance and revenue administrations interviewed during the country consultations. There is considerable interest of both the PICs and donors in strengthening budget and revenue systems because of the past history of unsustainable public finances in some countries and relatively poor accountability and lack of economic results despite the relatively high levels of donor support modality. Improved fiscal accountability is needed for donors to increase their use of the budget support which will increase the demand for PFM assistance. Reforms associated with regional trade liberalization initiatives will shift revenue away from customs and which is likely to increase demand for technical support in implementing tax reforms and improving tax administration.

92.	PFM and Revenue Administration activities were appropriately focused. PFTAC received in-depth assessments of the fiscal administration and policy priorities in the 8 PICs that are members of the IMF based on Article IV consultation reports. Both the PFM and Revenue Advisors were “in the information loop” when the IMF sent Headquarters missions to those countries. Increased awareness of the quality of PFTAC’s advice has led to the PICs becoming increasingly open in sharing their needs in the PFM and Revenue Administration areas. By exchanging information with the donors, PIFS and FEM, PFTAC has sharpened its focus in the fiscal areas. 

D.	Effectiveness of the Fiscal Assistance

93.	The effectiveness of PFM and Revenue Administration assistance were both rated as Good (see Table III.3). There was Good use of TA outputs and PFTAC coordinated well with other donors in the fiscal areas. Consistency with IMF Headquarters was rated as Excellent. Between 80% and 85% of the 70 survey respondents from the ministries of finance and revenue administrations rated the effectiveness of PFTAC in achieving results as Good/Excellent (see Table C.16).  Despite the generally positive views, respondents most directly involved with PFM and Revenue assistance indicated that there was room to improve effectiveness. Between a quarter and a third of those respondents assigned a Modest rating on this criteria (see Tables D.4 and D.5). Feedback indicated that more regional workshops and attachments, more support for regional approaches, improved follow up for implementation of fiscal reform programs and better monitoring and evaluation would improve the effectiveness of the fiscal assistance.

Table III.3: Effectiveness of PFTAC’s Fiscal Assistance
(Note: This information replaces a table that is available in the PDF version of this report)
Input of Resident Advisors (Person Months)
Weight1: N/A per cent; Ratings2—Public Financial Management: 36; Revenue/Customs Administration: 32; Weighted Total: 68
Use of TA outputs
Weight1: 50 per cent; Ratings2—Public Financial Management: 3.0; Revenue/Customs Administration: 3.0; Weighted Total: 3.0
Coordination with development partners and support for regional approaches
Weight1: 30 per cent; Ratings2—Public Financial Management: 3.0; Revenue/Customs Administration: 3.0; Weighted Total: 3.0
Consistency with IMF headquarters activities
Weight1: 20 per cent; Ratings2—Public Financial Management: 4.0; Revenue/Customs Administration: 4.0; Weighted Total: 4.0
Total rating
Weight1: 100 per cent; Ratings2—Public Financial Management: 3.2; Revenue/Customs Administration: 3.2; Weighted Total: 3.2
1 Column weights were defined by the Evaluation Team and row weights are based on the proportion of person months of Resident Advisor input for each group of TA activities.
2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor
Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

94.	Feedback from the country consultations also suggests that while the PFM and Revenue Administration assistance have made progress, more improvements are needed to achieve the outcomes set out in the Project Document, particularly for capacity building and developing fiscal and customs frameworks to support regional integration and free trade. The Fiscal Advisors both used short term experts, regional workshops and training to extend PFTAC’s ability to meet the demand for fiscal services. The importance of extending the range of PFTAC’s fiscal assistance is aptly summed up by a comment made by one of the survey respondents “The work of PFTAC should be supported with more resources to ensure that it delivers outputs that raise the level of public financial management in the region. Funding agencies pour a lot of resources into public auditing but little in financial management and is like putting the "cart before the horse...”

95.	Survey respondents directly involved in planning and implementing PFTAC’s financial assistance rated the use of outputs and implementing recommendations as Good but below the midpoint in the range for both PFM and Revenue Administration (see Tables D.4 and D.5).  The Evaluation Team identified examples in which the PFM and Revenue Administration outputs were well used in Fiji and Samoa and the following positive feedback was received from the survey “The Cook Islands has worked very closely with PFTAC on developing a medium term budgeting framework. Although the initial desire came from the Cook Islands, PFTAC has provided ongoing technical support to progress the development of this framework. The lead and ownership is taken by the Cook Islands. I am extremely happy with the support of PFTAC over the length of this project, their responsiveness and turnaround time is very efficient and effective. I do hope that their office remains and is given the credit it deserves through this survey.” PFTAC contributed to developing the Financial Manual in the Solomon Islands. However that experience illustrates that because of limited institutional capacity, it sometimes takes several years and considerable support from other donors before PFTAC’s recommendations can be effectively implemented. 

96.	Regional workshops organized by PIFMA and PITAA were effective vehicles for expanding the reach and effectiveness of skills transfer in the PFM and Revenue Administration areas. Feedback from both the field consultations and the survey found that identifying relevant, practical workshop topics, the quality of resource persons and opportunities for interaction with other participants doing similar professional work contributing to the success of the workshops.

97.	There has been an increasing exchange of information and collaboration in the PFM and Revenue Administration areas between PFTAC and other TA providers. New Zealand, Australia and the European Union are placing increasing emphasis on PFM because of a desire to make greater use of budget support in delivering their aid programs in the Pacific. Doing so requires stronger systems to manage public finances. ADB has provided considerable policy based lending and associated TA that is designed to support policy reform, including in the areas of PFM and revenue administration. While PFTAC’s expertise and quality assurance are widely respected, there is a need for donor coordination that goes beyond sharing information and work plans. PFTAC’s strengths are providing short term, rapid advice and undertaking diagnostics and quality assurance. Other donors can finance the long term assistance needed to support the implementation of major reforms, something that is beyond PFTAC’s resource envelop. The Evaluation Team identified several excellent examples of synergies between PFTAC and other donors that drew on the comparative strengths of the parties involved: (i) ADB and the Revenue Advisor worked together in Tuvalu with ADB providing a substantial TA and the Revenue Advisor helping to scope out the work, review its progress and arrange for officials from Tuvalu to attend workshops. AUSAID financed the computer systems needed to implement the recommendations; (ii) ADB requested the PFM Advisor to review several concept papers to help scope out the TA and to review and comment on draft reports; (iii) ADB used PFTAC’s diagnostic work and recruited a former PFTAC short term expert under a Cook Islands TA to follow on from earlier PFTAC work; (iv) ADB and PFTAC worked together in the GFS area in Vanuatu; and (v) AUSAID has drawn on PFTAC’s expertise in tax administration to help scope out work. Going forward, maximum use should be made of this type of engagement with other donors. If it works as was intended, the recent stationing of a World Bank staff in PFTAC with responsibility for PEFA should bring the PFM interventions of the two organizations closer together. 

98.	PFTAC fiscal TAs were highly consistent with the IMF Headquarters activities because of close operational links, the development of the APD regional strategy notes based on inputs from the PFM and Revenue Advisors and involvement in the preparations for, and follow up to, Article IV consultations in the fiscal areas. However, there is room to further improve the scope and details of the APD regional strategy notes to make them a more effective tool in the medium term planning of the fiscal assistance. This is particularly important given that most PICs that are IMF members are on a 24 month consultation cycle.

E.	Efficiency of the Fiscal Assistance

99.	Efficiency was assessed as Excellent for PFM and as Good for Revenue Administration. Both clusters were rated as Excellent for process and implementation efficiency because of the creative and flexible approaches taken by the Advisors and FAD’s efficient backstopping. PFM received an Excellent rating for the use of resources when factoring in the total level of activities from the PFTAC budget and additional donor funding the PFM Advisor is supporting. The PFM Advisors were the most successful among all Resident Advisors in making efforts to mobilize additional resources and use short term consultants to supplement PFTAC’s limited resources to meet the growing demand for its assistance. The approach of the PFM Advisors in mobilizing additional resources represents best practice and should be emulated by the other Resident Advisors.  The large majority of the survey respondents from fiscal agencies assigned a Good rating to PFTAC’s implementation efficiency with about 30% giving an Excellent rating (see Table C.31). Survey responses from those directly involved with PFM and Revenue assistance showed that over 90% assigned an Excellent/Good Efficiency rating. This result was confirmed during the country consultations which indicated that PFTAC’s fiscal TAs were efficient in terms of process implementation and value for money. Most respondents gave a Good rating to PFTAC’s timely response with 20% to 25% assigning an Excellent rating.
Table III.4: Efficiency of PFTAC’s Fiscal Assistance
(Note: This information replaces a table that is available in the PDF version of this report)
Input of Resident Advisors (Person Months)
Weight1: N/A per cent; Ratings2—Public Financial Management: 36; Revenue/Customs Administration: 32; Weighted Total: 68
Process and implementation efficiency
Weight1: 40 per cent; Ratings2—Public Financial Management: 4.0; Revenue/Customs Administration: 4.0; Weighted Total: 4.0
Efficient use of resources
Weight1: 40 per cent; Ratings2—Public Financial Management: 4.0; Revenue/Customs Administration: 3.0; Weighted Total: 3.5
Monitoring and reporting
Weight1: 20 per cent; Ratings2—Public Financial Management: 2.0; Revenue/Customs Administration: 2.0; Weighted Total: 2.0
Total rating
Weight1: 100 per cent; Ratings2—Public Financial Management: 3.6; Revenue/Customs Administration: 3.2; Weighted Total: 3.4
1 Column weights were defined by the Evaluation Team and row weights are based on the proportion of person months of Resident Advisor input for each group of TA activities.
2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor
Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

100.	Based on the cost per person month to deliver services, the fiscal assistance was considered efficient compared to other functional areas in PFTAC and other RTACs. Timeliness of responses and processes has been the hallmark of PFTAC’s operations compared to other TA providers serving the PICs. This was confirmed in field interviews and by the responses to the survey which identified rapid response as an area of PFTAC’s comparative advantage. The timeliness of PFTAC’s processes reflects its simple, lean operational structure. 

101.	The recruitment of Fiscal Advisors was managed efficiently with no long vacancies. While there were some delays in recruitment, there were no major adverse impacts on overall efficiency because there was always at least one advisor in the fiscal area, supported by FAD backstopping, to cover any urgent requests. Best practice was evident in overlaps with predecessors which minimized disruptions associated with changes in personnel. The PFM Advisor has made regular visits to Washington which allowed for consultations with FAD and APD.

102.	The use of short term experts to complement the work of the Resident Advisors has been a key feature for improving PFTAC’s efficiency and cost effectiveness in the fiscal areas, extended the reach and coverage of PFTAC’s fiscal support and leveraging the expertise of the Fiscal Advisors. The increased use of short term experts led to an increasing managerial role for the Advisors. As a rule of thumb, the Advisors indicated that at any point of time they could manage up to four short term experts working simultaneously and up to nine short term expert missions per annum. To reduce the administrative workload involved in recruitment and supervision, the Advisors used a pool of short term experts who had a proven track record and were familiar with the format and systems of reporting. The process of recruiting short term experts has been streamlined to an average of 10 days for mobilization from the day of appointment. Feedback from the field consultations acknowledged the efficiency advantages of using a pool of proven short term experts, subject to a stringent and transparent performance evaluation system involving both PFTAC and the beneficiary agency.

103.	The reporting of the PFM and Revenue Advisors included the standard back to office reports, monthly and quarterly reports to FAD and a regular flow of email and phone communications between Suva and Washington as part of the backstopping system. The weaknesses in the monitoring and evaluation of the achievement of development results reported in Chapter II also apply in the fiscal areas. The use of TAIMs as an online TA management system, when fully developed, is expected to improve the overall efficiency of TA management and reporting on the achievement of results. However, at this point in time the Fiscal Advisors do not see filling in information in TAIMS as adding value. Although formal self evaluation surveys were not undertaken on a regular basis, the network developed by the Fiscal Advisors allowed them to receive regular feedback from the PICs, particularly at the PITAA and PIFMA meetings.

F.	Sustainability of the Fiscal Assistance

104.	The survey respondents rated the sustainability of the benefits resulting from the PFM and Revenue Administration activities as Good but the Evaluation Team rated their sustainability as only Modest (see Tables III.5 and C.35). Concerns about institutional weaknesses and the limited use of Pacific expertise represent potential threats to sustainability. A substantial portion rated PFM sustainability as Modest which was consistent with 43% reporting that the implementation of PFTAC’s recommendations in this area was Modest (see Tables.D.4 and D.5). 

Table III.5: Sustainability of PFTAC’s Fiscal Assistance
(Note: This information replaces a table that is available in the PDF version of this report)
Input of Resident Advisors (Person Months)
Weight1: N/A per cent; Ratings2—Public Financial Management: 36; Revenue/Customs Administration: 32; Weighted Total: 68
Institutional absorptive capacity
Weight1: 40 per cent; Ratings2—Public Financial Management: 2.0; Revenue/Customs Administration: 2.0; Weighted Total: 2.0
Sustainable use of outputs
Weight1: 40 per cent; Ratings2—Public Financial Management: 3.0; Revenue/Customs Administration: 3.0; Weighted Total: 3.0
Promoting use of Pacific expertise
Weight1: 20 per cent; Ratings2—Public Financial Management: 2.0; Revenue/Customs Administration: 2.0; Weighted Total: 2.0
Total rating
Weight1: 100 per cent; Ratings2—Public Financial Management: 2.4; Revenue/Customs Administration: 2.4; Weighted Total: 4.4
1 Column weights were defined by the Evaluation Team and row weights are based on the proportion of person months of Resident Advisor input for each group of TA activities.
2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

105.	The Evaluation Team did identify examples of fiscal TAs that resulted in tangible and lasting benefits with outputs embedded in country fiscal systems. In Fiji, the skills transferred to Ministry of Finance staff in performance budgeting were used to train staff from other government ministries in the preparation of corporate plans and performance budgeting formats. The production of the handbooks and related training workshops helped in sustaining the outputs in the cash management and medium term fiscal frameworks for PFM and the management of large tax payers in the Revenue Administration area. PITAA and PIFMA regional workshops were well attended by the appropriate officials. Feedback from the survey showed that the workshops for both clusters of fiscal TAs provided knowledge that was used on the job by the participants. The trainees valued the practical applications discussed in the workshops and extensive use of examples from the Pacific Region.

106.	The evaluation identified some difficulties implementing TA recommendations, something that is critical to sustainability. This situation sometimes arose because PICs received general advice with limited practical application or the recommended solutions were not consistent with the capacity of the country. In some cases the resources needed to implement the recommendations were lacking although there were cases in which the PFM Advisors successfully worked with other donors that provided sometimes substantial resources to support the implementation of recommendations. The Revenue Advisor reported that he made it his mission to focus on the “how” phase of implementing recommendations. This was reflected in the content of the workshops and production of handbooks for revenue administrators. It was also necessary to adapt the outputs, particularly those activities promoting international best practices, to reflect the Pacific realities to improve the use of the fiscal outputs. 

107.	Developing the capacity of the PICs to absorb and use the outputs of the PFM and Revenue Administration assistance is a long term challenge. Feedback from the survey and interviews in the field identified high staff turnover and loss of trained staff as key factors that adversely affect sustainability. Even Samoa and Fiji, considered to have been relatively successful in embedding outputs in their budget and revenue systems, are concerned about the sustainability of their achievements given their thin pool of skilled personnel and risk of staff turnover. 

108.	Taking a long term approach to building capacities in the public finance sectors, PFTAC worked in partnership with the School of Governance and Development Studies at the University of the South Pacific to develop a course on Public Financial Management. The course focuses on basic public financial management in small developing economies. The course targets graduate students in the governance program and civil servants. Students are taught the principles of good budgeting systems and public financial management and examine the issues involved in budgetary and financial management reforms and effective prudential supervision of financial institutions including the banking system, provident funds and other trust funds. PFTAC contributed to the development of course material which is still used. The PFTAC Coordinator, PFM Advisors and Financial Supervision Advisor provide guest lecturers each year. The program has been offered annually since 2005. To date, 103 students have graduated or are currently enrolled. While the large majority (80%) is from Fiji, there have also been students from other PICs (see Table D.3). This is an excellent initiative as it addresses a critical constraint by helping to build a pool of qualified people in the PFM area. To ensure its continuing relevance, the course should be reviewed as a number of new concepts and practical tools have been introduced in the PFM area since the course was designed. Possible inclusion of a component on revenue and customs administration could also be explored.

109.	Interviews with senior officials and survey responses strongly favored PFTAC using regional experts. Although the use of regional experts in the fiscal areas was limited, the quality of the regional experts actually used was found by the Advisors to be good. In the PFM area, regional experts were used for the production of the Handbook on Containing the Civil Service Wage Bill in Pacific Islands and to assess the internal audit situation in the Pacific Region. The PFM and Revenue Advisors both sought to interest regional experts through their contacts in the PICs, but feedback was limited. Regional experts were used on a regular basis to make presentations at the PITAA and PIFMA workshops. For example at the 2008 PITAA meeting, the Fiji Inland Revenue and Customs Administration made a presentation on Regional Opportunities for Tax Administration in the Pacific. At the 2008 PIFMA meeting, the Samoan Ministry of Finance made a presentation on the experience of Samoa with the use of forward estimates while Nauru, Niue, Palau, Solomon Islands and Tuvalu delegations gave presentations on financial management systems and internal audit. In the PFM area, regional attachments have been piloted whereby regional personnel are attached to the PFM Advisor. Regional attachments were supported by AUSAID which funded the cost for the Fijian Inland Revenue and Customs Administration of accepting attachments from other PICs. 

110.	While PIFMA and PITAA have been valuable as vehicles for improving the relevance and effectiveness of PFTAC in the PFM and Revenue Administration areas, their long term financial sustainability needs to be addressed. For PITAA, a Tactical Plan: 2009-2011 paper was prepared to initiate discussions on the long term sustainability of the association. Consideration could be given to using the OCO model for PITAA whereby there is a twinning arrangement between OCO and the World Customs Organization for technical support and funding involves contributions from member PICs.

G.	 Lessons for the Future for Fiscal Assistance

111.	The evaluation identified a number of lessons in the fiscal area:
(i)	Increased Demand: Increased resources are needed to cope with the growing demand for fiscal assistance. A second PFM advisor is needed. One PFM Advisor could focus on providing rapid response services and on coordinating with other TA providers to support long term PFM reforms and capacity building activities.
(ii)	PFM Focus: Greater focus is needed to bring demand and supply into a better balance. Based on the evaluation findings, focus in the PFM area should be on: (a) cash management; (b) performance based budgeting; (c) supporting the undertaking of PEFAs and the resulting PFM action plans; and, (c) supporting the implementation of priority actions identified in the PFM action plans.
(iii)	Revenue Administration Focus: In Revenue Administration the focus should be on (a) supporting action plans for implementation of tax policy reforms, (b) preparing operational manuals and systems with related training to embed tax reforms into the systems of the revenue agencies; and, (c) supporting implementation of revenue reforms to facilitate regional trade reforms.
(iv)	Donor Collaboration: Closer collaboration with donors to use PEFA to develop a common framework for PFM reforms. 
(v)	Financial Sustainability of the Regional Associations: The efforts to identify mechanisms to ensure the long term financial sustainability of PITAA and PIFMA need to be continued.
(vi)	Internal Audit: The FY2006/08 Project Document suggested that PFTAC should provide assistance to strengthen internal audit, something that was identified in the Pacific Plan and the 2008 regional review of the internal auditing. Because of concerns about PFTAC’s and FAD’s depth of technical expertise in this area, the advisability of PFTAC becoming deeply involved in internal audit should be reassessed.

IV. ASSESSMENT OF FINANCIAL SECTOR SUPERVISION ASSISTANCE

Key Messages

•	The Evaluation Team rated the success of PFTAC’s portfolio of Financial Sector Supervision (FSS) assistance to be Good, as were the four categories evaluated: relevance, effectiveness, efficiency and sustainability. 
•	Within the categories, Excellent ratings were given in assistance in defining priorities, consistency with IMF Headquarters activities and process and implementation efficiency. There was Modest progress in monitoring and reporting and in the use of Pacific Island expertise.
•	Supervisory authorities as a group give roughly equal priority to TA in the supervision of banks, insurance companies, and superannuation funds, but most of PFTAC’s assistance has been in the area of banking supervision. MCM and PFTAC should consider ways to continue to seek out and deliver TA in the latter areas. In particular, superannuation funds may pose systemic risks for many countries in the Region, and they are often lightly supervised.

•	While PFTAC-funded short-term experts were not used for FSS, the demand for such experts may increase as the Fund is charging for IMF TA. 

•	MCM should provide PFTAC with copies of back to office reports and Plans for visits to PFTAC countries. The FSS Advisor should be given the opportunity to comment on the terms of reference and aide-memoires for MCM missions to PICs.

•	While the ad hoc TA needs assessment missions in FSS are useful, a more systematic method of identifying priorities is needed. Financial Sector Assessment Programs (FSAPs) and FSAP updates work well, but they are not feasible for most PICs. One approach would be to have annual needs assessment missions to perhaps three countries in sequence, with a team composed of the FSS Advisor and one or two short-term experts.	
A.	
Introduction to the Financial Sector Supervision Assistance

112.	In the area of Financial Sector Supervision (FSS), the FY2006/08 Project Document states that PFTAC will provide assistance in the following areas. At the national level, PFTAC will: (i) advise, in coordination with MCM and the Legal Department, on reform of bank and non-bank laws and regulations, and the organization of the financial supervision function; (ii) strengthen supervisory policies and processes, in the areas of licensing, on-site examination, and off-site surveillance; (iii) promote the adoption of adequate capital standards and effective risk management practices; (iv) promote closer cooperation and coordinate information sharing on financial regulation and supervision; (v) support Anti Money Laundering/Combating the Financing of Terrorism (AML/CFT) activities;  and, (vi) help streamline reporting systems to facilitate the collection and analysis of financial and prudential data from financial institutions. Planned regional activities included: (i) preparing, in conjunction with MCM, a summary of financial supervision arrangements within the region; (ii) helping PICs to compile Financial Soundness Indicators (FSIs); (iii) acting as secretariat for the Association of Financial Supervisors of Pacific Countries (AFSPC); (iv) engaging the Basel Committee on Banking Supervision and coordinating with the Financial Stability Institute; and (v) developing a module based financial supervision training course with the relevant training material placed on PFTAC’s web site. A review of the work done during this period shows that the FSS Advisor closely followed his terms-of-reference.

113.	During FY2006/08, PFTAC delivered 35 missions by the FSS Advisor, sponsored a series of workshops and seminars on financial sector supervisory topics attended by 95 participants, acted as the secretariat of the AFSPC, and arranged for eight attachments of FSS officials, five of whom were from FSM (see Table E.1).

114.	The distribution of FSS missions was balanced. The major recipients were the larger PFTAC countries, such as Fiji, the Solomon Islands and Vanuatu, and smaller countries that had particularly strong demands for TA (e.g., Cook Islands, FSM, Palau). Minimal assistance was provided to the countries with little or no need for such TA, including Kiribati, Nauru, Niue, Tokelau, and Tuvalu. The pattern for workshops roughly matched that for TA, except two of the larger countries, PNG and Samoa, made relatively little use of TA visits during FY2006/08, but sent a considerable number of staff to the workshops. 

115.	The Evaluation Team found that the current and previous FSS Advisors were pressed to their limits trying to service so many countries. Both averaged over one mission a month after allowing for holidays. On the surface, it appeared that some of the strains on the FSS Advisor could have been relieved by making more use of short-term experts. While the previous FSS Advisors reported that there was no excess demand for TA in FSS, discussions with the authorities revealed unmet demands, particularly in the areas of insurance and superannuation fund supervision. The previous PFTAC Coordinator and one MCM back stopper also noted that the FSS Advisors preferred to provide TA on their own, turning to MCM when additional resources were needed. Beyond this, the demand for TA in FSS is rising, and the expected shift to charging for IMF TA could also cause PFTAC member countries to demand the free TA resources PFTAC can provide. The former PFTAC Coordinator indicated that resources for short term experts were available but that in the FSS area they were not requested.

116.	When asked about their top priority for future TA in FSS, the supervisory authorities interviewed by the Evaluation Team were evenly split between banking, insurance, and superannuation fund supervision. One central bank governor described superannuation funds as a largely overlooked systemic risk in the region. While most of PFTAC’s assistance has been in the area of bank supervision, MCM has provided assistance related to the supervision of superannuation funds in Fiji and PNG and PFTAC has provided some assistance in this area in the Solomon Islands. In the insurance area, PFTAC provided assistance to FSM. The range of expertise required should be considered as one of the criteria when selecting FSS Advisors. Given the breadth of financial sector topics covered in PFTAC’s Project Document, the FSS Advisor must have the ability and willingness to cover many general aspects of the supervision of insurance and superannuation funds, as well as bank supervision. If the FSS Advisor does not have the knowledge and experience to handle the more technical issues, the requests for such assistance should be met by using Short Term Experts or missions from MCM, if need be financed with PFTAC’s budgetary resources. 

117.	All short-term expert assistance to PFTAC member countries was provided by MCM. This differs from other RTACs, which often spend significant resources on FSS experts. The PFTAC/MCM model for providing short-term assistance to PFTAC member countries will probably need to change, since not only does there appear to be an excess demand for FSS TA in the region, but because MCM’s TA resources have been under increasing pressure, while the demand for PFTAC help in banking supervision, and insurance and superannuation fund oversight is likely to rise.

B.	Rating the Financial Sector Supervision Assistance

118.	The Evaluation Team rated the success of FSS TAs as Good (see Table IV.1). These findings and those for the subcategories are broadly consistent with the responses to the PFTAC Evaluation Survey (see Tables C.11, C.16, C.31 and C.35 in Annex C and Table E.2 in Appendix E). The rational for these ratings is given in the following sections.

Table IV.1: Rating the Financial Sector Supervision Assistance
(Note: This information replaces a table that is available in the PDF version of this report)
Relevance
Weight: 20 per cent; Rating: 3.1
Effectiveness
Weight: 40 per cent; Rating: 3.2
Efficiency
Weight: 20 per cent; Rating: 3.2
Sustainability
Weight: 20 per cent; Rating: 2.8
Total rating
Weight: 100 per cent; Rating: 3.1
Column weights were defined by the Evaluation Team 
4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

C.	Relevance of Financial Sector Supervision Assistance

119.	The Evaluation Team rated the relevancy of the FSS assistance as Good (see Table IV.2). Consistency with Government priorities was Good, in some cases Excellent. Most authorities reported in their discussions with the Evaluation Team that FSS assistance was closely aligned with their priorities. Survey respondents gave a Good rating on consistency of PFTAC TA with government objectives, well above the midpoint on the Good scale (see Table C.11). The assistance was demand-driven. Several authorities, including those of the Reserve Bank of Fiji and the Central Bank of the Solomon Islands, reported that informal discussions with the FSS Advisors by phone and E-mail were particularly valuable. Experience has shown that discussions with Resident Advisors in the RTACs tend to be far more frequent and wide-ranging than those with IMF Headquarters.  

Table IV.2: Relevance of PFTAC financial sector supervision assistance
(Note: This information replaces a table that is available in the PDF version of this report)
1. Consistency with government priorities
Weight1: 50 per cent; Rating/score2: 3.0
2. Defining priorities
Weight1: 10 per cent; Rating/score2: 4.0
3. Quality of TA formulation and engagement
Weight1: 40 per cent; Rating/score2: 3.0
Overall relevance rating/score
Weight1: 100 per cent; Rating/score2: 3.1
Total rating
Weight1: 100 per cent; Ratings2—Public Financial Management: 2.4; Revenue/Customs Administration: 2.4; Weighted Total: 4.4
1 Weights defined by the Evaluation Team 
2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

120.	As one respondent to the survey stated the focus on supervision was appropriate “PFTAC's small resource capability, at least in the financial sector area, is appropriate given the small size of the financial systems of PFTAC member countries and that lack of financial markets. It thus makes sense to focus primarily on supervision-related issues and less on other areas.”  However, one concern is whether FSS TA is now fully aligned with both the needs of its members and the areas of expertise stated in the Project Document. While assistance for bank supervision should continue, as provided for in the Project Document the range of services should be broadened to provide TA in the supervision of insurance and superannuation funds. The authorities indicated they attached priority to TA in these latter areas. Several respondents noted that banking supervision in the region is better than insurance supervision and that superannuation fund supervision is weaker still. 

121.	PFTAC did an excellent job in helping countries to define their TA priorities in FSS. The authorities reported that when they asked for assistance, the FSS Advisors played important roles in helping them to define their priorities. Four of the seven authorities the Evaluation Team interviewed (i.e., FSM, Fiji, Solomon Islands, and Tonga) observed that PFTAC provided important help in this area. A number of countries also were assisted in defining their priorities by PFTAC needs assessment missions, which the FSS Advisors often tied into their initial missions to a country. For example, the FSS Advisor conducted diagnostic reviews in Samoa (2005), the FSM (2006), and Nauru (2006).  
 
122.	The quality of individual TA formulation and engagement was Good, and many thought they were Excellent. The authorities generally reported that the FSS TA was of very high quality, and that it was well adapted to their needs. Several authorities noted the value of the FSS Advisors’ cross-country experience, particularly with other countries in the region. 

D.	Effectiveness of the Financial Sector Supervision Assistance

123.	The effectiveness of FSS assistance was found to be Good (see Table IV.3). Good, and in some cases Excellent, use was made of the TA provided. Examples of TAs that were praised by the authorities as being particularly effective are given below.  Survey respondents from central banks rated the effectiveness of TAs in achieving results as Good (see Table C.16). Those that gave a detailed assessment on the survey of the FSS assistance rated the use of the TA outputs, practicality of the recommendations, the degree to which the recommendations were implemented and effectiveness of PFTAC contributing to capacity building as Good, but the proportion of respondents rating the use of outputs and implementation of recommendations as Modest exceeded those that rated these criteria as Excellent (see Table E.2). This suggests that there is room for improvement in these areas.

Table IV.3: Effectiveness of PFTAC Financial sector supervision assistance
(Note: This information replaces a table that is available in the PDF version of this report)

1. Use of TA outputs
Weight1: 50 per cent; Rating2: 3.0
2. Coordination with development partners and support for regional approaches
Weight1: 30 per cent; Rating2: 3.0
3. Consistency with IMF headquarters activities
Weight1: 20 per cent; Rating2: 4.0
Overall effectiveness rating/score
Weight1: 100 per cent; Rating2: 3.2
1 Weights defined by the Evaluation Team.
 2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

124.	There are a number of examples of successful TA interventions that were fully consistent with the authorities’ objectives and have resulted in significant changes in the authorities’ operations. In AML/CFT, PFTAC helped the Cook Islands Financial Supervisory Commission to tighten bank licensing laws because of concerns about money laundering, and PFTAC helped the Marshall Islands in getting removed from the Financial Action Task Force blacklist. In the Solomon Islands PFTAC helped the central bank to develop their capacity from scratch to implement their AML law. In banking supervision, five countries (e.g., FSM, Marshall Islands, Solomon Islands, Tonga and Vanuatu) reported that PFTAC was instrumental in helping them develop and implement improved prudential regulations and reporting systems. The very effective banking supervision TAs generally involved three or more missions, in some cases over a period of four years or more. In several of these cases, the authorities reported that the work was done jointly by MCM and PFTAC, with seamless cooperation. In Tonga, MCM and PFTAC combined to build from nothing what the authorities viewed as an effective banking supervision system over a period of four years. 

125.	Examples of PFTAC’s other FSS assistance covered a range of areas, including: (i) recommendations for strengthening the supervision of superannuation funds in the Cook Islands, Samoa and the Solomon Islands and insurance supervision in Samoa, the Solomon Islands and Vanuatu; (ii) assisting in on-site examinations in banking supervision, in the Cook Islands and Tuvalu, and in AML/CFT in the Solomon Islands; (iii) providing comments on draft legislation on banking supervision in FSM and Palau, and (iv) helping to draft guidelines in banking supervision, AML/CFT in Fiji and Niue.

126.	Discussions revealed that MCM backstopping was less intensive than that of FAD and STA. This difference was at least partly because of the three restructurings of MCM in the past six years, and the associated staff reductions that have diluted MCM’s capacity for backstopping. However, in practice, MCM’s backstopping could be less intensive because much of the TA is based on widely accepted standards and codes. In addition, the FSS Advisors indicated that although backstopping of most TA was minimal, strong backstopping support was provided on request. This contributed to TA quality, effectiveness and achieving results. 

127.	Survey respondents rated the FSS support for regional initiatives as Good (see Table E.2 was “owned” by the countries). The AFSPC, which is supported by PFTAC being its permanent secretariat, has done much to enhance regional cooperation among financial sector supervisors, as well as serving as a source of training, a forum for information sharing and networking and a starting point for regional initiatives. Respondents report that the material covered in the workshops was often used in their day-to-day operations. AFSPC workshops were so well received that a semi-annual training workshop was added in 2007.  In 2007, at the request of its membership, the AFSPC, through its Secretariat, began an effort to increase financial literacy in the region by circulating documents from the Basel Committee on Banking Supervision to AFSPC members and by acting as an e-library for the related documents. The 2006 AFSPC Workshop on FSIs, which was conducted with the assistance of IMF Headquarters staff, served as a basis for PFTAC member countries agreeing to report their FSIs on PFTAC’s webpage. On the negative side, several respondents noted that workshop follow-up was not always good.  

128.	The coordination of FSS assistance with other development partners was Good although, in practice not many donors are involved in bank supervision. For example, PFTAC cooperated with the Bank for International Settlement in delivering several workshops. A review of back-to-office reports to donors also showed efforts were made to ensure coordination. Five reports discussed coordination with specific donors, mostly with AMLAT and PALP on AML/CFT, and, in one case, coordination with NZAID on insurance legislation. 

129.	Most FSS assistance was provided on a bilateral basis. Although the Pacific Islands Forum’s Pacific Plan states that “Development of common approaches to financial regulation, including through alignment of legislation and/or pursuit of common prudential capacities,” this objective has not yet been achieved. While acknowledging PFTAC’s efforts to work with the Pacific Island Forum on this initiative, the Evaluation Team believes that PFTAC could have pursued this more vigorously both with the Pacific Island’s Forum, and elsewhere, notably through the AFSPC. While similar approaches across countries may have been favored by the FSS Advisors, it was not stressed at the operational level. The main area where regional approaches were encouraged was in workshops, particularly those associated with the AFSPC. Two examples of regional efforts are the workshops on supervising superannuation funds in 2005 and on FSIs in 2006. The former was followed up in 2007 with a PFTAC handbook on superannuation fund supervision. This handbook, although of good quality, has apparently not been widely used in the Region. The FSI workshop resulted in an agreement to publish members’ indicators on the PFTAC’s webpage. 

130.	The information on FSS on the web pages of PFTAC and AFSPC is good, and helps to enhance donor coordination, information sharing and outreach. There are several aspects that are worthy of note. On the positive side, the ASFPC web page contains two sections not contained in the other association web pages, a summary of regulatory regimes in the region and an area where PFTAC member countries may voluntarily post their FSIs. On the negative side, not all of the links from PFTAC’s web page to AFSPC’s web page worked all of the time, while a number of the links from the AFSPC web page section on papers and publications to documents, including all of the survey documents, did not work. Despite PFTAC follow ups, the section on FSIs was not always up to date, largely due to the voluntary nature of the postings.

131.	PFTAC assistance in FSS was highly consistent with IMF Headquarters activities. PFTAC was closely involved in the development of APD’s regional strategy notes for PFTAC member countries. This is all the more important now that most PFTAC member countries have been put on a 24-month Article IV consultation cycle. MCM closely coordinated with PFTAC in the development of its work program. A review of the Article IV Staff Reports and Public Information Notices (PINs) during FY2006/08 showed that when recommendations were made, they were generally followed up by PFTAC, MCM or another donor.  However, most PINs did not include recommendations for FSS that were within PFTAC’s terms of reference.  

E.	Efficiency of the Financial Sector Supervision Assistance

132.	Overall the efficiency of FSS TA FSS was rated as Good, above the midpoint in the range, with process and implementation efficiency being rated as Excellent (see Table IV.4). This rating is consistent with the efficiency rating given by central bank employees in the PFTAC Evaluation Survey (see Table C.31). The implementing agencies report that the TA was very timely, and a number of authorities noted that engagement was ongoing, even when there were significant periods between TA visits. Several authorities, including the central banks of Tonga and Vanuatu, and the FSM supervisory agency, spoke favorably of the close TA coordination between PFTAC and MCM. Recruitment of the replacement FSS Advisor was also done on a timely basis with no serious vacancies. 

Table IV.4: Efficiency of PFTAC financial sector supervision assistance
(Note: This information replaces a table that is available in the PDF version of this report)

1. Process/Implementation efficiency
Weight1: 40 per cent; Rating2: 4.0
2. Efficient use of resources
Weight1: 40 per cent; Rating2: 3.0
3. Monitoring and reporting
Weight1: 20 per cent; Rating2: 2.0
Overall efficiency rating/score
Weight1: 100 per cent; Rating2: 3.2
1 Weights defined by the Evaluation Team.
2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

133.	One issue that may undermine the efficiency of the TA is that MCM does not always share all the details of, and changes in, its TA plans with PFTAC.  While MCM provides its Regional Assistance Plan to APD and PFTAC, in the future the Regional Assistance Plans and updates should be sent to the FSS Advisor. Allowing the FSS Advisor to provide inputs and comment on terms-of-references and aide-memoires for MCM visits to PFTAC member countries would be useful and could help avoid potential problems. 

134.	Measuring the efficiency of the use of resources is difficult for all RTACs. Overall, it is believed to be Good, and the same appears to apply to FSS TA for PFTAC. The cost per person month of delivering FSS services compares favorably to the other functional areas in PFTAC and to the other RTACs (see Table A.6). However, it would be possible to increase PFTAC’s efficiency and to make more efficient use of the fixed overhead of the FSS Advisor’s remuneration if greater use were made of short term experts in FSS areas.

135.	As noted in Chapter II, monitoring and reporting on the achievement of development results is a problem throughout the RTACs and IMF more generally. While PFTAC has made some progress in results-based reporting for FSS, there are ongoing problems with the implementation of TAIMS, the envisaged centerpiece of this effort.

F.	Sustainability of the Financial Sector Supervision Assistance

136.	The sustainability of the benefits of FSS assistance was rated as Good (see Table IV.5), that of the survey respondents employed by central banks (see Table C.35) and people who gave a detailed rating of the FSS TAs (see Table E.2).. In part, this Good rating reflects the rather good organizational absorptive capacity in many central banks. However about a quarter of survey respondents who gave detailed ratings of FSS TAs had some concerns about sustainability and assigned a Modest rating. Organizational capacity varies considerably in region both between countries and institutions. Some of the smaller PICs find it hard to develop sustainable capacity, in part because of staff turnover. Most central banks and supervisory authorities were sanguine about their capacity and their ability to maintain capacity. This may be partly because central banks and supervisory authorities tend to pay somewhat higher salaries than other parts of government, which tends to reduce turnover and allow these institutions to hire higher quality employees. However, in some countries significant staff turnover reduced the sustainability of the benefits of PFTAC’s assistance. In some countries there are potential risks that the benefits of FSS assistance will not be sustainable.

Table IV.5: Sustainability of PFTAC financial sector supervision assistance
(Note: This information replaces a table that is available in the PDF version of this report)

1. Organizational absorptive capacity
Weight1: 40 per cent; Rating2: 3.0
2. Sustainable use of outputs
Weight1: 40 per cent; Rating2: 3.0
3. Promoting the use of Pacific expertise
Weight1: 20 per cent; Rating2: 2.0
Overall sustainability rating/score
Weight1: 100 per cent; Rating2: 2.8
1 Weights defined by the Evaluation Team.
2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

137.	The Evaluation Team found some evidence of the sustainable use of FSS assistance outputs. The interventions noted in the section above on Effectiveness on AML/CFT and financial sector supervision assistance resulted in new procedures and practices being introduced into the executing agencies. Another example is the implementation of the regional reporting system for FSIs. Most staff attending financial sector workshops reported that they used the material learned in their daily work. In addition, the Evaluation Team found strong evidence ownership of both FSS TAs and AFSPC. The Pacific Island Forum Secretariat has endorsed the ongoing capacity building efforts by PFTAC. 

138.	The majority of respondents with expertise in FSS agreed that the use of Pacific expertise is Important or Highly Important. However, PFTAC’s progress in this area has been Modest. The FSS Advisors were not from Pacific Islands nor were any short term experts. On the positive side, on three occasions a staff member from one PFTAC member country’s supervisory authority was attached to the supervisory authority of another member to provide support. One particularly successful example involved attaching a staff member of the Reserve Bank of Fiji to the National Reserve Bank of Tonga as the Acting Deputy Governor for Corporate Restructuring. Also, use has been made of regional presenters at seminars. 

G.	Lessons for the Future for Financial Sector Supervision Assistance

139.	A number of lessons for the future resulted from the evaluation of the FSS assistance:
(i)	Superannuation Funds: In many PICs these funds are the largest financial institution in the country. Despite their size, the funds are often lightly supervised, and may be a source of systemic risk. MCM/PFTAC should become more proactive and increase their supervision activities in this important area.
(ii)	More Diversified FSS TA: Discussions with the central banks and supervisory authorities showed that, as a group, they applied roughly equal priority to strengthening the supervision of banks, insurance companies and superannuation funds. MCM and PFTAC should consider ways to better seek out and deliver TA in the latter two areas, perhaps by hiring a short-term regional expert with PFTAC or mobilizing donor funds. 
(iii)	Evaluation of TA Needs: While the ad hoc FSS needs assessment missions have been useful, a more systematic method of identifying the TA priorities of PFTAC member countries would be useful. In larger countries, Financial Sector Assessment Programs (FSAPs) and FSAP updates work well. However, it would not be feasible to conduct FSAPs for most PFTAC member countries. One possibility would to have regular TA needs assessment missions to perhaps three countries in sequence, with a team composed of the FSS Advisor and one or two other short-term experts. Alternatively, one could consider regional missions on specific FSS topics such as the supervision of superannuation funds.
(iv)	 Short Term Advisors:  PFTAC should make greater use of short-term FSS experts to extend the coverage, reach and frequency of its assistance. Given that IMF is starting to charge clients of TA delivered by Headquarters but not for assistance provided through RTACs, demand for short term experts financed by PFTAC is expected to increase.
(v)	Length of Assignment for FSS Advisors: Experience in the island RTACs has shown that it takes six months to a year for a new Resident Advisor to be accepted by the clients and make his/her introductory visits to key client countries. After this period is completed and good relations have been built with key officials, requests for TA accelerate. The current system of using one year renewable contracts may create the impression of frequent turnover FSS Advisors that may undermine the effectiveness of PFTAC. As one respondent to the survey noted “It is highly recommended that advisors/consultants assigned to the Financial Sector Supervision area at PFTAC be allowed to continue their services more than the 3-year time-frame to enable much more sustainable advisory services to the countries.”
(vi)	PFTAC/MCM Coordination: MCM should provide PFTAC with back to office reports, and request that the FSS Advisor provide inputs and comments on terms-of-references for MCM missions and expert visits to PFTAC countries.

V. ASSESSMENT OF STATISTICS ASSISTANCE

Key Messages

•	The performance of PFTAC’s statistics assistance was rated as Good. The highest ratings were for effectiveness and efficiency with lower ratings for relevance (because of PFTAC’s over stretched model and lack of engagement at the strategic level) and sustainability (institutional weaknesses).
•	PFTAC would be more relevant if it were engaged in strategic issues (e.g., improving work planning and management; supporting the formulation of realistic, costed National Strategies for the Development of Statistics). 
 
•	Building capacity requires sustained engagement over a period of years. More than one visit per year is needed to build capacity. 

•	More resources are needed for short term consultants, another Statistics Advisor and the associated back stopping.
•	Given its limited resources, PFTAC should continue to focus its activities on national accounts and balance of payments. 

•	PFTAC should use the regional approach for statistics that it has successfully used in the other three functional areas by supporting the development of, and providing secretariat functions for, a regional macroeconomic statistics association or, alternatively, separate regional associations for national statistics organizations, central bank statistics and ministry of finance statistics. 

•	More synergies should be developed between the statistics assistance and PFTAC’s programs in the functional areas. 	

A.	Introduction to Statistics Assistance

140.	The Project Document states that at the national level in the area of economic and financial statistics, PFTAC will: (i) assist in developing national statistical strategies including the legal framework, institutional responsibilities and assessing resources required; (ii) advise on the regular and timely collection, compilation, analysis and dissemination of statistics drawing on the General Data Dissemination System (GDDS) and the Data Quality Assessment Framework (DQAF); (iii) improve human resources; (iv) pay special attention to the needs of smaller PICs; and (v) provide expert assistance to develop and maintain the basic macroeconomic series. At the regional level PFTAC planned to: (i) update the status of national statistical policies in the Pacific and prepare a strategy to improve the statistical systems in conjunction with STA; (ii) help improve dissemination and usage of statistics; and (iii) undertake annual seminars on the international standards and practices. PFTAC provides assistance to train staff, improve compilation techniques and bring statistics closer to international standards and produce good quality macroeconomic statistics. PFTAC does not finance the surveys necessary to improve the underlying data. 
 
141.	PFTAC’s statistics assistance was more narrowly focused than the ambitious plans laid out in the Project Document. Little PFTAC support was provided to address broad issues like improving the legal and institutional frameworks and assessing the resources needed for an appropriate statistical system. Regional workshops and training programs were not a major feature of PFTAC’s statistics assistance although in country training was provided as part of every mission. The Project Document was over ambitious relative to the resources provided to deliver statistics assistance.

142.	The Statistics Advisor undertook 35 missions during the three years covered by the evaluation (see Table F.1). This represents a very demanding travel schedule -- it is not reasonable to expect an Advisor to be able to travel more than once a month given the difficulty of travel in the Pacific Region. The Statistics Advisor provided assistance to central banks for monetary and financial statistics and balance of payments (BOP), national statistics offices for national accounts and price statistics and ministries of finance for government financial statistics (see Table F.2), with the greatest focus on national accounts and BOP.

143.	The work of the Statistics Advisors was complimented by 9 Short Term Expert missions and 9 attachments of Pacific Islanders. Combining the work of the Statistics Advisor and the short term experts, seven countries received an average of one of more visits per year over the three year period. Together these countries accounted for 80% of the statistics missions. Despite their relatively small size and the logistical challenges of travelling to the North Pacific, both FSM and the Marshall Islands received significant coverage. Given PFTAC’s resource constraints, role of other TA providers and the information in Table F.2, the geographic allocation of assistance and support for small countries appear reasonable.

B.	Rating the Statistics Assistance

144.	The performance of PTAC’s portfolio of statistics assistance was rated as Good. The statistics TAs were rated as effective in achieving results and were delivered efficiently – both of these dimensions of evaluation were rated as Good. Relevancy was rated on the border between Good and Modest and a Modest rating was assigned to sustainability. These lower ratings are not indicative of problems related to skills, dedication or competency of the Statistics Advisors. Rather they reflect: (i) a PFTAC model that requires the Statistics Advisor to cover multiple functional areas in numerous countries; (ii) limited missions to some countries – effective capacity building requires more than one visit per year to countries; and (iii) weak organizational absorptive capacity in the Pacific Region. 

Table V.1: Rating of statistics assistance
(Note: This information replaces a table that is available in the PDF version of this report)

Relevance
Weight: 20 per cent; Rating: 2.5
Effectiveness
Weight: 40 per cent; Rating: 3.0
Efficiency
Weight: 20 per cent; Rating: 3.2
Sustainability
Weight: 20 per cent; Rating: 2.0
Total rating
Weight: 100 per cent; Rating: 2.7
Note: Column weights were defined by the Evaluation Team.
4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

C.	Relevance of the Statistics Assistance

145.	Because senior government officials need accurate, reliable and timely statistics to formulate and implement sound macroeconomic policies, the Pacific Plan stresses the importance of upgrading and extending country and regional statistical information systems and databases. The IMF needs good quality macroeconomic data for its monitoring and surveillance work. Many stakeholders do not believe that the quality of Pacific statistics is adequate for sound macroeconomic management and effective monitoring of government and donor funded programs to determine the results being achieved. Problems relate to a limited range of statistics, coverage, reliability, timeliness, quality control, dissemination and inconsistencies with recognized international standards. The weakness of statistics, and the adverse impact on surveillance and policy making, is consistently referred to in the reports prepared by Article IV consultation missions. 

146.	The relevancy of the Statistics TAs was rated on the borderline between Good and Modest (see Table V.2). The Good rating for consistency with government priorities was offset by the Modest ratings for defining TA priorities and consistency of engagement. 

Table V.2: Relevance of PFTAC statistics assistance
(Note: This information replaces a table that is available in the PDF version of this report)

1. Consistency with government priorities
Weight1: 50 per cent; Rating/score2: 3.0
2. Defining priorities
Weight1: 10 per cent; Rating/score2: 2.0
3. Quality of TA formulation and engagement
Weight1: 40 per cent; Rating/score2: 2.0
Overall relevance rating/score
Weight1: 100 per cent; Rating/score2: 2.5
1 Weights defined by the Evaluation Team.
 2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

147.	The information in the World Bank’s Country Statistical Information Database was used to help judge the relevance of the statistics assistance. Ratings were available for 10 PFTAC members (see Table F.3). The average ranking for the strength of the statistical systems in PFTAC countries was 41 on a scale of 100, lower than the worldwide average of 65. No PIC exceeded the world average – the strongest countries scored around 50. The smaller countries received lower scores, less than half the world average. For statistical practice, the average score worldwide was 56 but it was considerably lower for the PICs at 32. Vanuatu scored slightly above the world average for statistical practice and Fiji, PNG, Samoa and Tonga received scores of between 40 and 50. The remaining countries received scores of 10 or 20.

148.	The data in Table F.3 demonstrates that there are macroeconomic statistical issues to be addressed in all PICs. Based on Table F.3, PFTAC’s support for macro-statistics is both needed and relevant. In 2002 PFTAC produced a comprehensive study, The Challenge of Statistical Capacity Building in the Pacific, which identified priorities for the PICs. The feedback received during the field interviews in Fiji, Samoa and the Solomon Islands and survey results confirmed the fact that PFTAC’s statistics assistance was well aligned to country needs. Survey respondents from statistical agencies and the respondents who provided a detailed assessment of PFTAC’s work in the statistics area ranked the consistency between PFTAC’s assistance and government priorities as Good (see Tables C.12 and F.5).

149.	While there are clear needs to strengthen statistics throughout the region, only one country, Vanuatu, has formulated a National Strategy for the Development of Statistics (NSDS). The process has been initiated in three other countries (PNG; Samoa; Solomon Islands). The Project Document envisioned PFTAC being involved in helping to strengthen legislation, developing such strategies and estimating the required resources. However, such work was not a major feature of PFTAC’s assistance. While respondents to the survey indicated that PFTAC played an important, and often a leading, role in defining TA priorities, this was at the technical level rather than at the strategic level. Other donors (e.g., Australia; New Zealand; ADB) have more funds than PFTAC to support statistics. In such circumstances it would be an appropriate for other donors to take the lead in supporting the preparation of NSDSs. However, PFTAC should advocate that NSDSs be prepared and support their preparation since PTFAC’s expertise in macroeconomic statistics is widely respected. 

150.	Statistical capacity is weak in the Pacific region. Addressing this problem requires taking a holistic approach to build institutional capacity on a country by country basis. If PFTAC is to champion this approach, it will require a different model. During the Evaluation Team’s fieldwork and in some of the replies in the comment section of the Survey, requests were made for more frequent and longer missions. According to data used to prepare PFTAC’s FY2010 budget, there were 203 budgeted person days of input for the Statistics Advisor and a demand for another 94 days, nearly half of the budgeted amount, which was unfunded. Regardless of how dedicated and hard working he is, one Statistics Advisor cannot have a significant impact on creating capacity in three different agencies in most of the 15 countries. PFTAC’s model of engagement, with its limited resources and an over stretched Statistics Advisor, undermines the potential relevance of PFTAC’s assistance. PFTAC’s current model of Resident Advisors primarily working on one country at a time is not a viable model if the objective is to have a significant capacity building impact in most of PFTAC’s client countries. If this is the goal, alternative strategies need to be developed and costed to achieve it.

D.	Effectiveness of the Statistics Assistance

151.	The effectiveness of PFTAC’s statistics assistance and each of the three effectiveness subcriteria were rated as Good (see Table V.3). This assessment is broadly consistent with the survey results. The respondents working in national statistics agencies rated PFTAC’s assistance in achieving the desired results as Good, bordering on Excellent (see Table C.16).

Table V.3: Effectiveness of PFTAC statistics assistance
(Note: This information replaces a table that is available in the PDF version of this report)

1. Use of TA outputs
Weight1: 50 per cent; Rating2: 3.0
2. Coordination with development partners and support for regional approaches
Weight1: 30 per cent; Rating2: 3.0
3. Consistency with IMF headquarters activities
Weight1: 20 per cent; Rating2: 3.0
Overall effectiveness rating/score
Weight1: 100 per cent; Rating2: 3.0
1 Weights defined by the Evaluation Team.
 2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

 
152.	Priority assistance areas for national accounts include: (i) improving the accuracy and timeliness of GDP estimates in current and constant terms; (ii) improving the coverage of areas such as the informal sector, remittances, tourism and investment; (iii) rebasing GDP estimates; (iii) improving GDP estimates by expenditure share and the production approach; and (iv) developing quarterly national accounts for some of the more advanced countries. The Evaluation Team’s fieldwork verified that PFTAC’s assistance has led to some improvements in the quality, coverage and timeliness of national account estimates in Fiji, Samoa and the Solomon Islands. These positive results reflected sustained engagement over a period of years. PFTAC’s self evaluation notes that there were improvements in the national accounts in Kiribati, Marshall Islands, Palau, Tonga and Vanuatu. There were, however, concerns about the sustainability of the benefits in FSM and Nauru rejected revised national accounts estimates prepared in conjunction with ADB. Officials in both Fiji and the Solomon Islands stated that the work of the Statistics Advisors in going through their national accounts work sheets line by line, reviewing questionnaires and offering advice on the spot for improvements was every effective and contributed to the publication of more accurate and more timely national accounts estimates.   

153.	The consumer price index (CPI) is the only price statistic produced in the Pacific Region. Although timeliness is generally reasonable, there is a need to rebase the CPI in many countries and there are compilation and classification issues. PFTAC limited support for price statistics is appropriate because: (i) SPC is providing assistance related to the rebasing of CPIs; and (ii) a Headquarters TA is providing assistance to develop an import price index.

154.	For BOP statistics, PFTAC helped to build staff capacity, improve data collection, better use administrative data, improve the methodology consistent with Balance of Payments Manual, fifth edition (BPM5) standards and reduce errors and omissions. The Evaluation Team confirmed that PFTAC’s assistance helped to improve capacity and BOP estimates in Fiji, Samoa, the Solomon Islands and Tonga. In other cases PFTAC’s BOP assistance did not lead to effective outputs (e.g., Marshall Islands). PFTAC’s self evaluation indicated that BOP TA resulted in improvements in PNG. One of the survey respondents stated that “The multisector advisor has done a good job in identifying the issues with respect to balance of payments statistics. He has also made excellent recommendations that are not always implemented.”

155.	PFTAC has provided limited assistance to improve government finance statistics (GFS). Solomon Islands authorities advised that they had not yet reached the stage where such assistance could be used effectively. PFTAC assistance contributed to some progress being made in the GFS area in Fiji and Samoa and PFTAC is supporting ongoing work in the Cook Islands. Progress in PFM to make the chart of accounts for budget classification consistent with GFS2001 will help to create conditions for GFS assistance to be requested and to be successful. Given that PFTAC and other donors are providing increasing levels of support for PFM, the demand for GFS assistance will likely increase in the future. Effectively servicing this demand will require good synergies between the PFM and Statistics Advisors. PFTAC has provided limited amounts of assistance for Monetary and Financial Statistics (MFS). 

156.	The results of the PFTAC Evaluation Survey were broadly consistent with the impressions gained by the Evaluation Team: (i) while the average rating was Good, more respondents ranked the use of TA outputs as Modest than Excellent; (ii) the practicality of the recommendations was rated as Good; and (iii) the degree to which recommendations were implemented was rated as Good but 30% gave a Modest rating on this criteria and there were very few Excellent ratings (3%) (see Table F.5). 

157.	The GDDS is a framework for assessing national statistical systems and promoting improved dissemination. Four countries have published the GDDS metadata (see Table F.3). Six PICs that are members of IMF have not subscribed to GDDS nor have the 5 non-IMF members. Although the terms of reference for the Statistics Advisor appointed in 2005 included managing a TA to support the adoption of GDDS, it was completed in 2006. Since then work on promoting GDDS was not a major feature the statistics assistance. Going forward, PFTAC should explore ways to help countries complete the process of subscribing to GDDS. 

158.	Australia, New Zealand and ADB all provide statistics TAs to the Pacific Region as does the Secretariat of the Pacific Commission (SPC). The funding available from IMF and PFTAC is modest compared to the resources of some TA providers. In the past the work of SPC focused on social and demographic statistics so there was a natural demarcation between PFTAC and SPC. However, recently SPC filled a macroeconomic statistician position vacant since 2005, so going forward there will be a need for enhanced coordination between SPC and PFTAC. PFTAC is aware of the work of other statistics TA providers (see Table F.2). The evaluation survey and interviews did not identify major problems related to donor coordination. Survey respondents rated PFTAC’s performance regarding coordination with other TA providers as Good (see Table F.5). Areas where improvements could be made include: (i) more proactively encouraging TA providers to support the preparation of NSDSs to help address some of the more strategic issues facing national statistics organizations; (ii) helping to scope out the terms of reference for longer term advisors to be funded by other organizations and offering to peer review their work; (iii) helping to mobilize funds from other organizations to undertake the surveys necessary to address fundamental weaknesses in the accuracy and timeliness of macroeconomic statistics; and (iv) working with other TA providers to try to ensure that assistance provided for measuring and monitoring poverty, the Millennium Development Goals, environment and gender do not divert scarce resources from the compilation and timely production of macroeconomic statistics. 

159.	The PFTAC model is designed to ensure consistency between the work of the Statistics Advisor and IMF Headquarters. Survey respondents rated the coordination between PFTAC and IMF Headquarters in the area of statistics as Good with about a fifth of the people giving an Excellent rating and very few giving a Modest or Poor rating (see Table F.5). Article IV staff reports inevitably point to weaknesses in macroeconomic data in the PICs and suggest that IMF and PFTAC should provide assistance to improve the quality of macroeconomic statistics. There was excellent consistency between PFTAC’s statistics assistance and those reports.

160.	One of PFTAC’s strengths is the use of STA back stoppers to review the work done by the Statistics Advisors to help improve TA quality. For FY2009, backstopping was budgeted to account for 9% of the time of the Statistics Advisor and 34% for STA staff allocated to the Pacific Region. The backstopping model is under stress. While much of the backstopping work falls on the Real Sector Division, backstopping in STA typically involves more people than in FAD or MCM. Generally one backstopper can support the PFM, Revenue and Financial Sector Supervision Advisors. However, reflecting the structure of the department, different backstoppers for the Statistics Advisor are assigned in STA for work related to statistics prerequisites and real sector, price, BOP, monetary and financial and GFS statistics.     

161.	IMF has stringent criteria before it publishes data in the International Financial Statistics (IFS). The key findings of a comparison of the information in Table F.4 from the October 2008 IFS edition with the September 2005 IFS edition were: (i) there remain issues related to coverage and timeliness; (ii) there were some improvements in the timeliness of BOP data in countries in which PFTAC has provided assistance (e.g., Fiji; PNG; Samoa; Tonga) but in other countries there were not significant improvements (e.g., Solomon Islands); (ii)  there were some improvements in the coverage and timeliness of national accounts data in countries in which PFTAC has provided assistance (e.g., Samoa) but in other countries there were no significant improvements; (iii) there was limited improvement in GFS coverage in the few countries in which PFTAC had provided support (e.g., Fiji; Samoa; Tonga). Progress in improving the timeliness and coverage of data in the IFS has been uneven and slow and gaps remain, which illustrate the challenges faced in the Pacific Region in improving statistics. 

E.	Efficiency of the Statistics Assistance

162.	The efficiency of PFTAC’s statistics assistance was rated as Good (see Table V.4). The Excellent rating of process/implementation efficiency reflects flexibility and rapid response to requests for assistance, the excellent expertise of the Statistics Advisors and the amount of time the Statistics Advisor spends in the field. All people interviewed commented positively on these aspects. Respondents from national statistics agencies also rated efficiency as Good (see Table C.31). 

163.	When there was a turnover of Statistics Advisors, the recruitment was handled efficiently. There were no long term vacancies in the position – rather only a few weeks. Both the former and current Statistics Advisors were invited to Washington for one week orientation briefings. Although there was no overlap or formal briefing/handover between the two advisors, the outgoing Statistics Advisor left hard copies of key documents and an end-of-assignment report. Any queries were covered by E-mail between the two advisors. Statistics Advisors submit regular reports to STA and their work is evaluated by STA on an annual basis as part of the contract extension decision. 

Table V.4: Efficiency of PFTAC statistics assistance
(Note: This information replaces a table that is available in the PDF version of this report)

1. Process/implementation efficiency
Weight1: 40 per cent; Rating2: 4.0
2. Efficient use of resources
Weight1: 40 per cent; Rating2: 3.0
3. Monitoring and reporting
Weight1: 20 per cent; Rating2: 2.0
Overall efficiency rating/score
Weight1: 100 per cent; Rating2: 3.2
1 Weights defined by the Evaluation Team.
 2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

164.	PFTAC is cost effective in delivering its statistics assistance. The cost for PFTAC to deliver a person month of statistics TA compares favorably to the PFTAC average and the corresponding figures for other RTACs (see Table A.6). There are, however, areas where PFTAC could increase its cost efficiency: (i) making more use of Short Term Experts; and (ii) organizing a regional association of macroeconomic statisticians that could develop compilation handbooks for regional best practice, quality assurance of source data and dissemination standards and present opportunities for learning from other countries and peer review. 

F.	Sustainability of the Statistics Assistance

165.	The sustainability of the benefits of the statistics assistance was rated as Modest (see Table V.5). A quarter of Survey Respondents gave Modest/Poor ratings for sustainability (see Tables C.35 and F.3).

Table V.5: Sustainability of PFTAC statistics assistance
(Note: This information replaces a table that is available in the PDF version of this report)

1. Institutional absorptive capacity
Weight1: 40 per cent; Rating2: 1.0
2. Sustainable use of outputs
Weight1: 40 per cent; Rating2: 3.0
3. Promoting the use of Pacific expertise
Weight1: 20 per cent; Rating2: 2.0
Overall sustainability rating/score
Weight1: 100 per cent; Rating2: 2.0
1 Weights defined by the Evaluation Team.
 2 Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poor   Excellent ≥ 3.5; 3.5 < Good ≥2.5; 2.5 < Modest ≥ 1.5; Poor < 1.5.
Source: 2009 PFTAC Evaluation

166.	The capacity in the national statistical systems in the Pacific Region varies widely. In some countries there is reasonable capacity and in others it is weak (see Table F.2). In general statistical agencies are institutionally weaker than central banks, ministries of finance and revenue administrations. Because of organizational weaknesses and limited absorptive capacity there are significant challenges regarding the sustainability of benefits. Major problems include: (i) weaknesses in management, strategic prioritization and work planning; (ii) a lack of qualified staff, staff turnover and vacancies,, partly because of low salaries resulting from staff in statistics agencies being classified at a lower level than staff in other organizations; (iii) poor documentation which aggravates the problem of staff turnover and sustainability; (iv) very small statistics offices in some countries; (vi) sometimes staff are diverted to other priorities when donors finance special projects; and (vii) less than optimal use of administrative data. 

167.	As provided for in the Pacific Plan and as requested by FEM, a major study is being undertaken for the Pacific Islands Forum Secretariat and SPC that will identify the present capacity of statistical services and recommend options to strengthen the collection, analysis, dissemination and utilization of statistics in the region. The study will be completed in 2009 but based on an informal review of the draft report its conclusions regarding institutional capacity will not differ significantly from the 2002 PFTAC report or the conclusions summarized above and evident from Tables F.2 and F.3. The respondents in the PFTAC Survey who were most knowledgeable about statistics generally either agreed or strongly agreed that the sustainability of the benefits of PFTAC’s assistance was undermined by staff turnover, staff shortages and budget shortages. These issues need to be addressed at a broader, more strategic level than is possible with the type of TA that PFTAC has provided in the past.  

168.	The large majority of the Survey Respondents with knowledge of statistics stated that it was either Highly Important or Important for PFTAC to promote the use of qualified Pacific expertise. Only once during the three year period did PFTAC use a Pacific Island statistician as a short term expert. More opportunities should be sought to use Pacific expertise. PFTAC’s self evaluation indicates that in the statistics area while attachments were popular, there were mixed results. Placements with Statistics New Zealand were generally successful but attachments at PFTAC were less successful because of issues related to the selection process, lack of clearly defined outputs, limited mentoring and limited self-motivation on the part of the candidates. As a result of these issues, PFTAC suspended the attachment program in statistics program for a year and changed the selection process to direct nomination by PFTAC, with input on the candidate’s suitability by bilateral partners and SPC, and refocused the attachments on compilation. 

169.	A lack of well qualified staff with degrees in statistics is a major problem in the Region.  To help address this problem PFTAC worked with the University of the South Pacific to develop a course in statistics. This on-going initiative will help to address one of the key threats to sustainability. A PFTAC Statistics Advisor was one of the authors of a paper that describes the basis and rational for the course. The program, which began in 2005, is designed for practitioners and focuses on issues related to national statistical systems, empirical analysis, data collection and the interaction of statistics agencies with other organizations. The course can be taken on the university campus or extramurally. PFTAC was instrumental in helping to establish this course and several students from the statistics course were attached to PFTAC to gain applied experience. Although the current and former Statistics Advisors offered to give guest lecturers, these did not materialize. The Statistics Advisors encourage counterparts to enroll in the course and some staff from countries other than Fiji have begun to do so.    

G.	Lessons for the Future for Statistics Assistance

170.	Lessons for the future for PFTAC’s work in statistics include:
(i)	More Resources: No matter how dedicated and hard working, one Statistics Advisor cannot have a serious impact on capacity development when he tries to support national accounts, price statistics, BOP, MFS and GFS statistics in multiple countries. To increase the relevancy and sustainability of PFTAC’s statistics assistance, additional resources are needed for more short term consultants, another advisor and the associated back stopping. 
(ii)	Address Strategic Issues: In many countries, institutional weaknesses and limited budgets undermine the sustainability of benefits. Addressing these weaknesses requires a more strategic approach that helps to improve work planning and management, makes greater use of administrative data and supports the formulation of realistic NSDSs for which the required resources are costed and mobilized from the donor community. 
(iii)	Longer Periods of Engagement: Building capacity requires sustained engagement over a period of years. If the Statistics Advisor is only able to visit a country for one mission a year, the impact on capacity building will be limited.
(iv)	Develop a Regional Approach: PFTAC should use the regional approach for statistics that it has successfully used in the other three functional areas whereby it provides the secretariat functions for a macro-statisticians association. In addition to providing a forum for networking, sharing country experiences, peer review and regional training, such an organization would help PFTAC to identify and address areas of common interest, deliver regional training courses and identify common approaches for groups of countries. The latter should help to refine PFTAC’s statistics work program and lead to economies of scale. In commenting on the draft report, PFTAC noted that managing such a regional association would be difficult because it would involve statisticians from three organizations (i. e., national statistics agencies; central banks; ministries of finance) from 15 countries. Consequently it was believed that setting up three organizations would be a better way to proceed. From the Evaluation Team’s perspective, either approach would represent progress in the right direction. Cost savings could be realized by synchronizing such meetings with SPC meetings of the heads of statistics offices, regional workshops and the use of internet based discussion groups. 
(v)	 Develop Synergies with the Other Functional Areas: More opportunities should be sought to develop synergies between the statistics assistance and the other function areas (e.g., PFM and GFS; financial sector supervision and MFS and BOP; and tax administration/customs and trade and GFS).
(vi)	Intensify the Hands on Approach: The hands on approach where by the Statistics Advisor helps the statisticians to refine the calculations of national accounts and BOP estimates is effective and should continue so that there is learning by doing and at the end of the mission more accurate, up to date estimates are available.
(vii)	Need for Focus: Given the limited resources, there is a need to continue to focus and narrow the range of activities supported by PFTAC. The two priority areas should continue to be national accounts and BOP. However, as the work on PFM in the region progresses there will be a growing demand for GFS TA. 

VI. RECOMMENDATIONS

•	More follow up and diligence will be needed by all members of the PFTAC governance structure to implement the recommendations in this evaluation so that they will have more influence than the 2004 evaluation appears to have had.

•	Recommendation 1: By the end calendar year 2010, TPRC and IMF should develop a strategic plan that sets out a vision for where PFTAC should be in five years time. Scaling up the successful PFTAC model would be consistent with the findings of the evaluation. This would involve increasing the number of resident advisors (see Recommendation 2), making greater use of short term experts (Recommendation 3), increasing the chance that the benefits of PFTAC’s assistance will be sustainable (Recommendation 4), making greater use of Pacific expertise (Recommendation 5) and intensifying the use of regional approaches (Recommendation 6). Broad agreement on the way forward should be reached at the June 2009 TPRC meeting.

•	Recommendation 2: By the end of FY2010, consensus should be reached at TPRC on the priorities for additional Resident Advisors. Based on its work, the Evaluation Team identified three priority areas for additional Resident Advisors: (i) Macroeconomic Advisor; (ii) second PFM Advisor; and (iii) second Statistics Advisor.

•	Recommendation 3: Beginning in FY2010 PFTAC should make greater use of short term experts to leverage the expertise and associated fixed costs of Resident Advisors, particularly in the financial sector supervision and statistics areas. 

•	Recommendation 4: By the end of FY2010 the TPRC, PFTAC Coordinator, Resident Advisors and the TA Departments should develop a strategy to strengthen the likely sustainability of the benefits of PFTAC assistance.  

•	Recommendation 5: Beginning in FY2010 PFTAC should make a concerted effort to develop and use Pacific expertise.

•	Recommendation 6: By the end of FY2011 TPRC and PFTAC should develop a strategy to intensify and extend the use of regional approaches to build capacity in the Pacific Region.

•	Recommendation 7: By the end of FY2011 PFTAC should, in coordination with the TA Departments, clearly define medium term objectives to be achieved in each functional area in each country and verifiable indicators against which progress can be monitored.

•	Recommendation 8: By the end FY2010 PFTAC should develop formal action plans for each recommendation accepted by the TPRC, identifying the necessary resources and monitorable benchmarks to implement those recommendations and report on the implementation status of the action plans to the TPRC in FY2011 and again in FY2012.	

A.	Implementation Status of the 2004 Evaluation Recommendations

171.	The Evaluation Team assessed the implementation record of the recommendations in the 2004 evaluation on a four point scale (see Annex G). Of the 24 recommendations, 10 were rated as Poor, 5 as Modest, 6 as Good and one as Excellent. The implementation status of two recommendations was not rated. The results of the 2004 evaluation and its recommendations are referred to in the FY2009/11 Project Document. While this is a positive finding, insufficient action was taken to implement the recommendations. It is a relatively common experience for aid agencies to take little notice of evaluation recommendations. Evaluation results are not the only things to go into making decisions on whether or not to take action. Other factors include the institutional priorities, the views of other parts of the organization, available budget and the influence of external parties. Also, evaluators often spend so much time undertaking the technical parts of evaluations that there is insufficient time to carefully craft the recommendations and build support for implementation. 

172.	The recommendations were largely addressed to the TPRC and PFTAC, a positive feature since they were the main audiences for the evaluation. However, the Evaluation Team believes that the formulation of the recommendations in the 2004 Evaluation may have contributed, in part, to the limited implementation: (i) there were a large number of recommendations, perhaps too many, that were not prioritized or sequenced; (ii) some of the recommendations were quite operational in nature and may have diverted attention from the more important strategic recommendations; (iv) some of the recommendations were more akin to lessons learned; (v) there was no action plan or estimate of the resources needed to implement the recommendations; and (vi) there was no timeline or clear accountability for taking the necessary action for each recommendation. More follow up and diligence will be needed by all members of the PFTAC governance structure to implement the recommendations in this evaluation so that it has more influence than the 2004 evaluation appears to have had. 

B.	Recommendations

173.	The Evaluation Team has limited the number of recommendations, focused them on issues that relate to addressing strategic issues that need to be considered if the PFTAC model is to be elevated to the next level. Issues at this level need to be considered by the TPRC and in the headquarters of the PICs and organizations providing the financial support for PFTAC. While the PFTAC Coordinator and OTM can make proposals, the necessary decisions must be undertaken at a higher level. Matters at the operational level were presented as lessons at the end of Chapters II through V rather than as formal recommendations. These should be considered and addressed by the PFTAC Coordinator, Resident Advisors and the IMF backstopping system.

174.	Deciding whether or not, and how, to address the recommendations requires inputs and views from many other actors (e.g., other parts of IMF; senior decision makers in donors; senior government officials). It was beyond the scope of the evaluation to build the consensus necessary to address such issues and to identify the best or most feasible approaches. For various reasons it may be a legitimate decision not to take action to implement some recommendations (e.g., people feel that the recommendation is either wrong or not feasible; there are other more pressing priorities; the timing is wrong; the necessary resources are not available). However, there should be a clear process to determine which evaluation recommendations are accepted and which are not. For those that are accepted a plan of action for implementation must be developed. Based on the evaluation findings, eight recommendations are proposed to provide input for the future operations of PFTAC and the drafting of the Project Document for the next funding cycle. 

175.	Recommendation 1: By the end calendar year 2010, TPRC and IMF should develop a strategic plan that sets out a vision for where PFTAC should be in five years time. There are two broad options and many variations in between. Option 1: Continue business as usual within roughly the same resource envelop. Under this scenario PFTAC would continue to deliver good quality services but much of the assistance would be ad hoc, PFTAC would have limited engagement in capacity building at the broad strategic level and PFTAC would continue to be able to meet only a fraction of the demand for its services. If Option 1 is selected, the objectives in the Project Document for the next funding cycle should be scaled back to be consistent with the likely resource envelop. Option 2: PFTAC’s resource envelop would be significantly increased to allow it to deepen its engagement in strategic areas and provide more services. This would involve increasing the number of resident advisors which must be complemented with additional resources for backstopping (see Recommendation 2), making greater use of short term experts (Recommendation 3), increasing the chance that the benefits of PFTAC’s assistance will be sustainable (Recommendation 4), making amore concerted effort to use Pacific expertise (Recommendation 5) and intensifying the use of regional approaches (Recommendation 6). Scaling up the successful PFTAC model would be consistent with feedback received by the Evaluation Team. One respondent’s comment summarizes these views “As PFTAC size is small, there is a need for more resources (staff) to accommodate the demands from Pacific Island countries.” If the second option is selected an action plan would need to be developed to map out the transition of PFTAC in its current form to the future vision. While it will take some time to develop the details associated with the future vision of a significantly expanded PFTAC, this process can only begin in earnest if some initial consensus is reached on the way forward at the June 2009 TPRC meeting.

176.	During the review of the draft report within IMF, some concern was expressed that scaling up PFTAC would undermine some of the characteristics that made PFTAC successful. These concerns generally fell into three categories: (i) PFTAC would lose its strengths of being a lean, cost effective organization with limited bureaucracy; (ii) PFTAC would begin to focus on long term implementation where other well resourced donors are already active; and (iii) the demands that an expanded PFTAC would place on the backstopping systems of TA departments. The Evaluation Team believes that Option 2 could be developed in ways that satisfactorily address these issues:
(i)	Lean Organization: If three additional Advisors were added to PFTAC’s staff compliment, PFTAC would be about the same size as West and East AFRITAC. Those organizations share many of the same characteristics as PFTAC – they are lean and efficient. If all the Evaluation Team’s recommendations were implemented, PFTAC would remain a relatively small operation. While an increased use of short term experts would require Resident Advisors to do more backstopping, the Evaluation Team notes that while greater use was made of short term experts in the fiscal area than in the other functional areas, the number of missions undertaken by the Resident Advisors did not differ significantly across the functional areas. 
(ii)	Focus on Long Term Implementation: One of the findings of the evaluation was that PFTAC needs to provide more support for the implementation of its recommendations. Some additional resources would allow PFTAC to provide more assistance in this area. However, even if PFTAC receives all of the additional resources recommended by the Evaluation Team, it still would not have the resources necessary to provide the level of TA that some others can provide to finance comprehensive long term capacity building assistance in a country. PFTAC would need to continue to help mobilize assistance from other donors by identifying priorities, scoping out the required assistance and peer reviewing the resulting work. The Evaluation Team notes that while both East and West AFRITAC have 7/8 resident advisors, they provide similar services as does PFTAC and are not in competition with other donors to provide large teams to undertake capacity building in particular countries.  
(iii)	Backstopping: Increasing the number of Advisors would need to be complimented with additional resources for backstopping in the TA departments. Given the new system that IMF has introduced to record the use of staff time in greater detail to allow donors to be charged for the actual backstopping costs, it should be possible to satisfactorily address this issue.  

177.	Recommendation 2: By the end of FY2010, consensus should be reached at TPRC on the priorities for additional Resident Advisors. Based on its work, the Evaluation Team identified three priority areas for additional Resident Advisors that could be considered for Option 2: (i) Macroeconomic Advisor; (ii) second PFM Advisor; and (iii) second Statistics Advisor.

1.	Macroeconomic Advisor

178.	The description of PFTAC did not include the provision of macroeconomic support as one of its core functions. The absence of a macroeconomist among PFTAC’s advisors is surprising given that PFTAC’s overall mission is to support training and capacity building in economic and financial management and the fact that the PICs look to IMF for macroeconomic advice and economic forecasts, particularly during times of economic turbulence. This gap reflects (i) the internal structure of IMF’s technical assistance departments, which mirrors the functions of the four PFTAC Advisors; and (ii) the delineation between technical and policy issues. In the first case, IMF’s TA departments provide backstopping and oversight for fiscal matters, financial sector supervision, and statistics. IMF’s area departments conduct macroeconomic analysis. They were not established to backstop TA advisors. In the second case, PFTAC was established to provide technical advice. It was argued that the provision of policy advice might duplicate the IMF’s head office functions during the Article IV missions. IMF operates its regional centers as extensions of its own operations, rather than as stand-alone projects, which helps explain PFTAC’s lack of macroeconomic support. 

179.	Most PICs have a limited capacity to formulate sound macroeconomic policies. Generally the countries visited would welcome help to build their capacity to prepare macroeconomic forecasts and to undertake financial programming exercises. PFTAC should move towards fulfilling its mission by recruiting a Macroeconomic Advisor. In the words of one survey respondent “More in the area of economic and financial markets should be done by PFTAC, including experts at PFTAC and PFTAC to coordinate formation of a regional association for south pacific economists”. To avoid conflict with IMF’s policy formulation objectives the Macroeconomic Advisor, who would be backstopped by APD, should focus on the technical aspects of macroeconomic work: (i) preparing economic forecasts; (ii) helping the PICs prepare a macroeconomic framework; and (iii) assisting with financial programming. The Macroeconomic Advisor would help the PICs prepare for the Article IV missions to enable a more fruitful discussion, and assist with the implementation of the resulting recommendations. 

180.	The Macroeconomic Advisor would provide the PFTAC team with an anchor, synergy and direction in the other functional areas. In the view of the Evaluation Team, stationing a Macroeconomic Advisor in PFTAC is critical to the fulfillment of PFTAC’s mission. A respondent from one major donor stated on the survey that they “… would be keen to see PFTAC step up its support on macro-policy issues. Region still has need of a high quality locally-owned economic and fiscal policy advisory facility, and PFTAC could potentially evolve to fill this role”. This view was echoed by some senior government officials and representatives from the Pacific Islands Forum Secretariat who noted that PFTAC’s economic presentations to FEM meetings were very well received. Coordination with others will be required, particularly with ADB which approved a $3 million TA for Pacific Economic Management in April 2009 to help PICs improve economic management processes. The steering committee for that TA includes ADB, PFTAC and the Pacific Islands Forum Secretariat. With good cooperation, synergies could be found and the Macroeconomic Advisor could assist ADB in improving the national systems and quantification of macroeconomic analysis.

2.	Second PFM Advisor

181.	There is likely to be an increasing demand for PFM advice in the future. Because PICs are under increasing budgetary pressures, there is an increasing recognition of the need to set appropriate budget priorities and to execute the budget in a way that is consistent with the principles of transparency and accountability. Also some donors wish to increase their assistance to the PICs with general budget support. However, this is only possible if the donors have confidence in the way that the funds will be managed and spent. PFM TA helps to create the enabling conditions in which budget support can be provided, effectively managed and ultimately be viewed by both the government and the donors as being successful. All parties recognize the expertise of PFTAC and IMF more generally in the area of PFM and would welcome increased PFTAC involvement in helping governments to develop PFM action plans to set priorities and sequencing for improvements, help to scope the terms of reference for larger TA provided by other donors and provide peer review services. The current PFM Advisor is over stretched. PFTAC will only be able to cater to the increased demand if a second PFM Advisor position is created.

3.	Second Statistics Advisor

182.	In most countries, the organizational capacity of the national statistics agency is considerably weaker than that of the central bank and ministry of finance. There are issues related to management and strategic planning, budgetary allocations, the number and quality of staffing, staff turnover, data quality, coverage and timeliness and making the optimal use of administrative data. The Evaluation Team’s conclusions about the general weaknesses of institutional capacity in the statistics area are broadly consistent with the conclusions of PFTAC’s 2002 report on the challenges of building statistical capacity in the Pacific and the draft of the Pacific Islands Forum Secretariat’s forthcoming 2009 statistics benchmarking study. Many of the issues related to statistical capacity need to be addressed at a strategic level and would be embodied in an NDSD.  In most countries issues related to organization, management, budget, staffing and undertaking timely surveys will have a more direct bearing on building capacity than improving computational techniques. Currently the Statistics Advisor is expected to cover most of the PICs and generally three agencies in each country (e.g., national accounts and price statistics in the national statistics agency; monetary and financial statistics and BOP statistics in the central bank or financial supervision agency and government financial statistics in the ministry of finance). Regardless of how competent, hard working and dedicated the Statistical Advisor is, it is not possible for him to achieve sustainable impacts at a strategic level when working under such conditions. Focus and priority setting are needed. The Evaluation Team recommends that priority continues to be placed on national accounts and BOP. However, even if the focus of statistics support is narrowed, it is not possible for one Statistics Advisor to deliver TA that will achieve sustainable impacts under such circumstances when most countries receive only one mission per year. A second Statistics Advisor is needed to enable PFTAC to deepen its engagement at the country level and to help to address the more strategic issues related to building statistical capacity.

183.	Recommendation 3: Beginning in FY2010 PFTAC should make greater use of short term experts to leverage the expertise and associated fixed costs of Resident Advisors, particularly in the financial sector supervision and statistics areas.  The results of this evaluation confirm the findings of previous evaluations that PFTAC is filling a niche in the Pacific Region and is providing effective and efficient services that are very much appreciated by its clients. However, PFTAC’s ability to respond to the demand for its services is limited by its resources. PFTAC does not make as extensive use of short term experts as do other RTACs – the ratio of expenditures for short term experts to the salaries of Resident Advisors is 33% for PFTAC which compares to a range from 46% to 76% for the other RTACs. The survey results indicated that increasing the use of short term experts was an area in which PFTAC could improve and one respondent stated that “… increased budget resources should be given to PFTAC to enable it to employ short-term consultants to assist the resident adviser do his/her work as the workload is considerable given the number of member countries”. The need to increase the use of short term experts is not uniform across functional areas. Short term experts are used relatively extensively in the fiscal areas but much less so in the other functional areas. More short term experts should be used in the financial sector supervision and statistics areas. If the budget for short term experts is increased, there must be a corresponding increase in the resources for backstopping. Some of this burden will fall on the TA departments, particularly for highly technical areas. However, the Evaluation Team believes that the Resident Advisors have the capacity to do most of the backstopping for short term experts in their areas of expertise. The constraints on increasing the use of short term experts should largely be driven by PFTAC’s resources, both financial and the time of the Resident Advisors, rather than Headquarters’ resources in the Regional Assistance Plans.   

184.	Recommendation 4: By the end of FY2010 the TPRC, PFTAC Coordinator, Resident Advisors and the TA Departments should develop a strategy to strengthen the likely sustainability of the benefits of PFTAC assistance. The lowest ratings of the Evaluation Team concern sustainability. Contributing reasons related to the small size and limited institutional capacity and budgets of many of PFTAC’s clients and a loss of trained people. The loss of one or two key staff can have a major adverse impact on the capacity of small organizations. Addressing sustainability issues is a long term, challenging task as many of the factors relate to institutional problems that are endemic in the Pacific Region. Achieving success in this area will depend, in part, on actions taken by governments and the donor community. However, there are things that PFTAC can do to enhance the potential sustainability of the benefits of its assistance. More follow up to support the implementation of TA recommendations was identified in the PFTAC Evaluation Survey as an area where PFTAC could improve. All parties involved in PFTAC’s governance structure must do some strategic thinking on how to improve the sustainability of the benefits of PFTAC assistance. Options to consider as part of a strategy to increase the probability of sustainability include: 
(i)	More follow up from Resident Advisors and short term experts to support the implementation of TA recommendations, although given that all PFTAC Advisors are already over stretched and cannot reasonably be expected to undertake more missions, this may be difficult to accomplish within the current PFTAC model;
(ii)	Ensuring that all TAs clearly define and cost the resources needed to implement the resulting recommendations, develop a strategy to secure the necessary financing from other donors and government commitments to take the actions necessary to support the implementation of the recommendations, identify the counterparts who will be responsible for overseeing the implementation of the recommendations, estimate the government’s budgetary contributions and develop a reporting system to monitor and track progress;
(iii)	Vigorous coordination with donors by PFTAC and the concerned country to mobilize financing for a team of long term consultants, necessary computer hardware and software, long term training and possibly financing necessary surveys for statistics if such assistance is needed – PFTAC could work with the interested donor to help to scope out the necessary work and offer to peer review or monitor its implementation; and,
(iv)	To better cope with the loss of trained staff, the beneficiary agencies should be encouraged to effectively document policies and procedures, undertake succession planning and plan knowledge and skills transfer among staff. 

185.	Recommendation 5: Beginning in FY2010 PFTAC should make a concerted effort to develop and use Pacific expertise. The evaluation found that although the majority of stakeholders agree that PFTAC should develop and use Pacific expertise, PFTAC made only modest use of Pacific expertise. Survey respondents gave this topic the lowest rating among all of the factors assessed. It is important to stress that PFTAC should keep its high technical standards and only use Pacific Islanders if they have the appropriate experience and expertise. However, the few cases in which Pacific experts were used resulted in positive outcomes. PFTAC should be more pro-active in identifying and employing people from the region as short term experts. As a first step PFTAC should develop a roster of qualified and experienced people who could fulfill this role. Initially they could be used as resource persons for attachments and presenters at regional workshop and eventually as short term experts. However, PFTAC should raise its sights in this area and consider how to implement the recommendation in the 2004 Evaluation of developing a second tier of advisors in PFTAC that would be from the region. Such people could be in mid-career and be seconded to PFTAC for two to three years. They would work under the guidance of the Resident Advisors and would focus primarily on supporting the implementation of TA recommendations in their area of expertise and disseminating knowledge across the PICs about successful approaches. This approach would help build expertise in the region and provide a means of strengthening sustainability by supporting the implementation of TA recommendations and disseminating information on best Pacific practice. In making this recommendation the Evaluation Team is aware that that the pool of expertise is small and that that strong management and backstopping would be required to ensure that TA quality does not erode. Because of the need for considerable mentoring by the concerned Resident Advisors, this concept could be pilot tested first in those areas that PFTAC receives a second Resident Advisor. Despite these challenges, the Evaluation Team believes that PFTAC could be more proactive in its efforts to use Pacific expertise. 

186.	Recommendation 6: By the end of FY2011 TPRC and PFTAC should develop a strategy to intensify and extend the use of regional approaches to build capacity in the Pacific Region. The evaluation found that PFTAC’s efforts to establish and support AFSPC, PIFMA and PITAA were successful and highly appreciated in the Region. These organizations provide a good forum to network among peers, share country experience and promote Pacific best practice. They also represent a cost effective way that PFTAC can deliver training, identify priorities for TA in functional areas and identify issues that are common across a group of countries. Budgetary allocations to for the support of AFSPC, PIFMA and PITAA were an effective and efficient way to achieve results. The most obvious extension of this approach would be for PFTAC to help establish a regional association for macro economic statistics modeled on the successful experience in the other functional areas. Doing so should involve close coordination with SPC. The Evaluation Team believes that there is an opportunity to broaden this regional approach to support horizontal capacity building and south/south cooperation. 

187.	A significant barrier to building sustainable capacity in many PICs is their small size. Most government agencies are correspondingly small with staff member having primary expertise in a given area with limited backup. The small size of these agencies means that many functions that are carried out by one or two people or sometimes even on a part time basis that in a larger country would be done by a group of staff. The small size of the organizations makes internal training and staff development difficult, since staff are thinly stretched and generally do not have the time to learn the work of others in their agency. Capacity in an area can be seriously weakened if one or two staff members leave.

188.	 This inability to develop self-sustaining capacity within an agency, which may be called a lack of vertical capacity, imposes rigidities in government and exposes TA to sustainability risks. Resources spent in developing capacity in an agency may be lost because one or two staff resign, transfer, are promoted or immigrate. If vertical capacity cannot be developed within an agency, then building horizontal capacity across countries should be considered. This would involve a group of countries working together to jointly achieve sustainable capacity in a particular function. This approach could make it possible to fill a temporary staffing gap in a PIC through attachments, secondments or training provided by another PIC. 

189.	The Caribbean Regional Technical Assistance Centre (CARTAC) used this approach in introducing value added taxes (VATs) in a number of its member countries. One country asked CARTAC to assist in introducing a VAT, and then other countries found that they needed a VAT and liked the model that CARTAC was proposing. The initial strategy in the pilot country was developed by a team led by IMF/FAD, which included the CARTAC Advisor on Tax and Customs. Subsequent TA was delivered largely by CARTAC, with strong FAD backstopping. As sufficient progress was made in one country, the implementation team moved on to the next country. At each stage, the team sought to identify staff from the earlier recipient countries to work as experts in the later recipient countries. 

190.	One promising way to identify appropriate topics for this approach would involve the professional associations which PFTAC helps manage. A key precondition for joining a regional project should be that the country agrees to follow an approach quite similar to that used by the other participants. The similarities should cover legislation, methodology and systems of implementation. This is necessary to ensure skills learned in one country are largely transferable to other countries in the group. Staff from the newer participants should be encouraged to engage in attachments to earlier participants, both to improve networking and to further deepen horizontal capacity. 

191.	Recommendation 7: By the end of FY2011 PFTAC should, in coordination with the TA Departments, clearly define medium term objectives to be achieved in each functional area in each country and verifiable indicators against which progress can be monitored. Previous evaluations and discussions at TPRC meetings have highlighted the need for PFTAC to do a better job at managing for development results, i.e., focusing more on the achievement of outputs and outcomes rather than reporting on inputs. The evaluation has found that this is an IMF-wide issue rather than a PFTAC specific issue. As an institution, IMF has devoted considerable resources to this issue but as in many other organizations this is an unfinished agenda. Despite the progress that has been made, the quality of reporting on results needs to be further improved across all functional areas. More meaningful tracking is required that emphasizes outcomes rather than the use of inputs and the delivery of outputs. The key issue in this area that is within the control of PFTAC and the TA departments is to better articulate measurable objectives that are to be achieved in the medium term rather than focusing on a one year time horizon to plan, implement and monitor TA. PFTAC recognizes that improving its country notes should be a priority and it has plans to take action in this area. 

192.	Recommendation 8: By the end FY2010 PFTAC should develop formal action plans for each recommendation accepted by the TPRC, identifying the necessary resources and monitorable benchmarks to implement those recommendations and report on the implementation status of the action plans to the TPRC in FY2011 and again in FY2012. The generally poor implementation status of the recommendations in the 2004 Evaluation suggests that periodic reporting to TPRC is needed to reinforce efforts to implement evaluation recommendations.