Annex B: Evaluation Approach and Methodology
I.	Objectives of the Evaluation
1.	Independent evaluations are an integral part of the PFTAC governance framework to assess PFTAC’s performance and to review the implementation of recommendations in the 2004 PFTAC evaluation. The evaluation primarily covers the period covered by the FY2006/08 (May 2005 – April 2008) funding cycle. The evaluation had four objectives:
(i)	Evaluate the TA provided by PFTAC for relevance, effectiveness, efficiency and sustainability;
(ii)	Assess whether PFTAC is the right size to achieve its objectives, or alternatively, if the objectives need to be adjusted; in this context, consider PFTAC’s position in the regional TA architecture and whether there may be opportunities to further build on the support to better meet Pacific island countries’ needs;
(iii)	Examine the effectiveness and  frequency of TPRC meetings, and explore alternate modalities for more frequent contact; and
(iv)	Compile a set of lessons that may be used to strengthen PFTAC operations.
2.	The evaluation analyzed and rated the TA program by functional area, assessed the strengths and weaknesses of the PFTAC model and identified lessons for the future. TA interventions were assessed using four dimensions of evaluation: 
(i)	Relevance, which covers issues such as:

a)	Whether PFTAC played a useful role in helping to define country TA priorities in line with best practice, diagnostic assessments and policy advice provided by IMF. 
b)	The degree that PFTAC assistance equipped countries with adequate institutional capacity to define their own policy alternatives.
c)	The extent to which TA met the priority needs of member countries including whether the TPRC was effective in ensuring country ownership of PFTAC assistance.
d)	Whether PFTAC activities were appropriately focused in terms of subject areas, taking into account the IMF’s expertise and the priority needs of PFTAC member countries.

(ii)	Effectiveness, which covers issues such as:

a)	Whether TA is focused on delivering outputs that contribute to outcomes consistent with best practice and the achievement of member country reform priorities. The evaluation also assessed, to the extent possible, the TA outcomes and impacts.
b)	Options for increasing PFTAC’s impact through regional responses to specific issues. 
c)	The appropriateness of the mix of services provided by PFTAC.
d)	The extent to which PFTAC activities were well coordinated and leveraged with those of other donors, TA providers and regional agencies. 
e)	The extent to which PFTAC activities were well integrated with the TA, surveillance and lending activities of IMF Headquarters, including whether PFTAC was effective as a complement to TA from IMF headquarters, and in supporting strategies and best practice determined by Headquarters.
f)	Given its limited resources, determine whether PFTAC was more effective in certain functional areas than others.

(iii)	Efficiency, which covers issues such as:

a)	Whether TA was cost-effective, especially in relation to other delivery modes (as determined by the evaluators), and bearing in mind the difficulties inherent in measuring the benefits of capacity-building activities. 
b)	The quality and timeliness of PFTAC activities and outputs produced and the quality, timeliness, modalities, cost of management and backstopping by IMF Headquarters. 
c)	The quality and timeliness of reporting and monitoring on PFTAC activities and outputs.

(iv)	Sustainability, which covers issues such as: 

a)	The extent to which PFTAC assistance has led to tangible and lasting results.
b)	Whether there are constraints faced by PFTAC member countries which have prevented them from taking full advantage of TA and how such constraints can be addressed. 
c)	The effectiveness of PFTAC in identifying, utilizing, and promoting growth of local expertise, including through the appropriate use of local and regional TA experts.
d)	The extent to which PFTAC will be financially sustainable.
3.	The evaluation was designed to identify key areas for potential improvement or change to provide guidance for the future by:

a)	Examining the size and mix of advisors for PFTAC, taking into consideration its, three-year planning horizon, IMF TA plans, demand for its services, the regional absorptive capacity, and long-term results;
b)	Considering opportunities for increasing the value of PFTAC through an analysis of niche areas where it is considered successful by donors and beneficiary countries;
c)	Assessing if the TPRC is effective and the appropriateness of the 18 month cycle for meetings. 
d)	Considering alternative modalities for maintaining close contact among TPRC members; 
e)	Reviewing PFTAC’s position in the regional architecture of TA support; and
f)	Compiling lessons from the experience of the last funding cycle.

II.	Evaluation Approach and Methodology

A.	Evaluation Criteria and Rating Methodology
4.	The terms of reference require the application of a quantitative rating methodology within an evaluation framework that is based on relevance, effectiveness, efficiency and sustainability. These four dimensions of evaluation are commonly used in the international evaluation community. The sub-criteria, rating methodology and weights applied are described below. The evaluation methodology draws heavily on the approach and methodology used for the recent evaluation of the AFRITACs and IMF’s TAs in Iraq, modified as appropriate to reflect the needs of this evaluation and the sub-criteria suggested in the Terms of Reference. This will be the third evaluation commissioned by OTM that explicitly specifies and rates sub-criteria for relevancy, effectiveness, efficiency and sustainability.
5.	The Evaluation Team believes that this type of evaluation methodology reflects best practice. The use of numerical ratings forces evaluators to come to a conclusion. While reaching such conclusions involves judgment, this type of evaluation methodology helps to make such judgments transparent to the reader. The approach used to derive the overall rating is described below. It involves two steps: (i) rating each of the sub-criteria on a four point scale of excellent (4 points), good (3 points), modest (2 points) and poor (1 point); and (ii) assigning weights to each of the sub-criteria, functional area and the four dimensions of evaluation so that the results can be aggregated to reach broader conclusions about PFTAC’s overall performance. For this evaluation, the following weights were assigned: (i) relevancy (20%); (ii) effectiveness (40%); (iii) efficiency (20%); and sustainability (20%). Many of PFTAC’s activities were ongoing works-in-progress so their sustainability was difficult to judge. These weights, as well as the weights described below for the sub-criteria, were subjective and reflect the best judgment of the Evaluation Team. They were disclosed in an approach and methodology paper that was prepared at the beginning of the assignment and was circulated to IMF staff in PFTAC, OTM, MCM, FAD, STA and APD. 
6.	PFTAC activities were grouped into the following clusters:
(i)	fiscal affairs covering both public financial management and revenue administration; 
(ii)	financial sector supervision; and, 
(iii)	statistics as one cluster covering macroeconomic statistics, central bank statistics and public finance statistics.

7.	The three broad areas – fiscal affairs, financial sector supervision and statistics – correspond to the three TA departments (FAD; MCM and STA). This correspondence facilitated the analysis of the complementarity of PFTAC activities with TA provided from Headquarters. For PFTAC, these clusters were rated against the specific sub-criteria for the four dimensions of evaluation. The scores for the clusters of activities were added together, weighted by the person months of inputs used to deliver each cluster of activities to provide the basis for evaluating PFTAC’s overall performance and performance in the functional areas. 
1.	Relevance
8.	The sub-criteria used to assess relevance are listed below with the weight applied shown in brackets:
(i)	Consistency with Government Priorities (50%): This evaluation sub-criteria assessed the extent to which PFTAC activities reflected the priorities of beneficiary countries. The evaluators looked for evidence of consistency between PFTAC TAs and government priorities in feedback from government officials, both during face to face interviews during the fieldwork and from the PFTAC evaluation survey and inferences drawn from a review of IMF documents. The role of the TPRC in fostering country ownership of the TAs and strategies was assessed. 
(ii)	Defining Priorities (10%): This sub-criteria assessed the degree to which PFTAC helped to define priorities.
(iii)	Quality of TA Formulation and Engagement (40%): This sub-criteria assessed the quality of the TA strategy and formulation and diagnostic assessment process including whether country level needs assessments were undertaken, the consistency of the objectives and the resource inputs, consistency of engagement over a period of years.

9.	Each relevancy sub-criteria was rated on the four point scale separately for each functional area. The ratings were then aggregated, using the weights assigned to each sub-criteria. The overall ratings were derived by aggregating the ratings for each cluster of activities using the assigned weights. Standard descriptors, based on numerical cutoffs, describe the relevance rating: 
(i)	Excellent. The weighted score was greater than 3.5. This rating was given to TAs whose relevancy ratings were particularly good on all of the sub-criteria. There is a high probability that the TA achievements will lead to results that will be sustainable and will remain highly relevant. Few TAs achieve such a high rating.
(ii)	Good. The weighted score was between 2.5 and 3.5. Although there were some issues related to one of more of the sub-criteria that prevented a highly relevant rating, there were no major shortfall, and the TA resulted in relevant outputs that are likely to be sustained. The TA remains relevant and the problems were small relative to the positive findings. 
(iii)	Modest. The weighted score was between 1.5 and 2.5 on a scale of 4. The evaluation identified shortfalls in one or more of the sub-criteria. There were some positive findings for the TA but not what was originally expected. 
(iv)	Poor. The weighted average was less than 1.5. The TA had clear problems at the time of evaluation and did not succeed in achieving development results.
 
10.	Similar descriptors were used for the other three dimensions of evaluation. This approach provided a consistent basis for the various assessments. The Evaluation Team reviewed and modified the initial rankings using an iterative process to ensure consistency between the functional areas. The relative rankings of the sub-criteria were also reviewed for logical consistency. Particular attention was given to those assessments where ratings near the cut off points used to define the descriptors. 
2.	Effectiveness
11.	The evaluation of effectiveness of PFTAC activities assessed the degree to which the stated objectives were achieved or can be reasonably expected to be achieved given the developments at the time of the evaluation. The analysis was structured to examine whether PFTAC was more effective in some functional areas than others. The sub-criteria, and the assigned weights, used to evaluate effectiveness were as follows: 
(i)	Use of TA Outputs (50%): This involved assessing the use of the outputs of each TA cluster and whether the outputs are leading, or are likely to lead, to the desired outcomes. Evidence was sought as to whether the policies and procedures of the executing agency were changed to incorporate TA recommendations. Even if PFTAC provided high quality advice and assistance, if the TA outputs were not being used, the desired outputs will not be achieved. In many cases, there was an unfinished agenda. In such cases, the Evaluation Team made a judgment as to whether the expected outcomes were likely to be achieved. In making judgments about what might happen in the future, the Evaluation Team used a relatively short timeframe, about a year, and assumed that the amount of additional resources needed to achieve the TA objectives was modest compared to the time and effort that had already been made.
(ii)	Coordination with Development Partners and Support for Regional Approaches (30%): The importance of donor coordination and support for regional initiatives were highlighted in the PFTAC Program Documents and the 2004 evaluation recommended that greater attention be paid to this area. The Evaluation Team assessed the extent to which PFTAC TAs were coordinated, and leveraged with, with those of other TA providers, the role that the PFTAC played in facilitating coordinating donor assistance and its support for regional approaches to common issues. 
(iii)	Consistency with IMF Headquarters Activities (20%): The extent to which PFTAC TAs were integrated with TA from IMF Headquarters, surveillance activities and Article IV consultations was assessed under this sub-criteria by drawing on the results obtained from the PFTAC evaluation survey, a review of documents and interviews with staff from APD and the TA Departments. 

3.	Efficiency
12.	The evaluation of efficiency assessed the performance of PFTAC converting resources (e.g., human resources/expertise; financial resources; time) to achieve outcomes and results. The efficiency sub-criteria and weights are described below: 
(i)	Process and Implementation Efficiency (40%): Assessing process and implementation efficiency covered such factors as IMF’s internal management of PFTAC activities and the quality and timeliness of management and backstopping by Headquarters-based staff. Another factor examined under the criteria was the consistency of engagement at the institutional and individual level for both PFTAC and the executing agencies. Other factors considered included the timely recruitment of qualified PFTAC Coordinators and Resident Advisors, the efficiency of planning and executing the TA delivery and the timeliness of TA execution.
(ii)	Efficient Use of Resources (40%): The data available from IMF’s old time recording and financial and management information systems did not link staff time to specific TA initiatives. Thus it was not possible to undertake a proper cost efficiency analysis to assess whether PFTAC assistance has proven to be cost-effective compared to other TA delivery modes. The available data did allow the Evaluation Team to compare the cost per person month of input across functional areas and to compare PFTAC to other RTACs but not to compare the cost per person month of input for Headquarters TA/training with PFTAC TAs/training. IMF’s new time recording system should provide the necessary data to permit a better cost efficiency analysis in due course. 
(iii)	Monitoring and Reporting (20%): The Evaluation Team examined the degree to which PFTAC used self evaluation (i.e. monitoring and client feedback) and better reporting to improve the efficiency and effectiveness of their activities. That included examining the use PFTAC made of TAIMS and to respond to requests to report on outputs rather than inputs and to practice the principles of managing for development results.

4.	Sustainability
13.	The sustainability relates to the extent to which TAs led to tangible and lasting results. Sustainability depends on whether the institutional, policy and legal framework and human and financial resources are sufficient to sustain TA benefits. The sub-criteria used to assess sustainability included: 
(i)	Institutional Absorptive Capacity (40%): In most countries the central bank is the strongest agency which reflects institutional autonomy and higher salaries compared to other agencies that facilitate the recruitment and retention of qualified staff. The ministry of finance is usually among the strongest of the line ministries but the minister of finance is only one of several ministers. Typically in PICs the national bureaus of statistics are smaller and have less stature and access to human and financial resources than do central banks and ministries of finance. In general a rating of “3” was given under this criteria for central banks, a “2” for ministries of finance and revenue authorities and a “1” for national statistics agencies. While there are clearly differences among the PICs in terms of institutional capacity, it was beyond the scope of the evaluation to rate the institutional absorptive capacity of each agency in each country. These general country ratings reflect the views of the Evaluation Team based on those countries visited during the field work, interviews with other officials, feedback received from the Resident Advisors and information in PFTAC’s country notes. 
(ii)	Sustainable Use of the Outputs (40%): An important indicator of sustainability was whether the TA outputs were embedded in the routine businesses practices of the executing agencies. Another important factor considered when assessing this sub-criteria was whether the workshop participants/trainees used the knowledge gained on the job. This sub-criteria also considered whether or not trained staff remain with the organization. The loss of trained staff undermines the sustainability of TA benefits.
(iii)	Promoting the Use of Pacific Expertise (20%): This sub-criteria examined the degree to which PFTAC identified, used, and promoted the growth of local expertise in their activities, including through the appropriate use of local and regional TA experts, contributes to sustainability. This sub-criteria was assessed during interviews of key informants, an analysis of the PFTAC Evaluation Survey and a review of information provided by PFTAC.

14.	The person months of Resident Advisors and Short Term Experts, number of seminar participants and number of attachments used to deliver PFTAC assistance for the three functional areas is given in Annexes D, E and F for FY2006/08.
5.	Aggregation of the Ratings for PFTAC
15.	The assessment of PFTAC’s performance was derived by aggregating the relevancy, efficiency, effectiveness and sustainability scores for each of the clusters of activities. The weights for the activity clusters reflect the person months of inputs required to deliver the services. The following weights were assigned: (i) fiscal – 60% (53% for PFM and 47% for Revenue/Customs Administration); (ii) Financial Sector Supervision – 21%; and (iii) statistics – 19%. 
III.	Field Work
16.	The criteria for selecting countries visited included: (i) the location of the PFTAC; (ii) level of PFTAC engagement; (iii) maximize the number of countries covered within budget and time constraints; (iv) attempt to visit at least one country in Melanesia, Micronesia and Polynesia; and (v) a range of countries by size and level of economic development. Feedback received from the PFTAC Coordinators on the suggested country visits was used to finalize the plan for the fieldwork. Meetings were held with government officials, donor representatives and PFTAC staff in the field and with concerned staff in Washington. 
17.	The time schedule was for all of the fieldwork to be completed March/April 2009. The schedule of the field work was as follows: (i) week of 30 March: visit IMF Headquarters in Washington; (ii) week of 6 April: PFTAC, government agencies and concerned donors in Suva; (iii) week of 13 April: the Evaluation Team was split into individual missions to visit Samoa and the Solomon Islands. Telephone interviews were undertaken with senior officials in the Federated States of Micronesia and the Marshall Islands. Discussions were held with senior officials from the Central Bank of Samoa, the Central Bank of the Solomon Islands, the National Reserve Bank of Tonga, and the Reserve Bank of Vanuatu on the sidelines of a central bank governors meeting in Nadi in Fiji.

IV.	Data Sources

A.	Triangulation of Information
18.	The Evaluation Team drew on information from many different sources including documents and systems data available from IMF and interviews with a wide range of key informants. To broaden the coverage of the evaluation an electronic survey was undertaken of senior government officials, workshop participants, IMF staff, short term experts, former IMF staff and representatives of other TA providers. In reaching its conclusions, the Evaluation Team considered evidence from several sources and attempted, to the extent possible, to validate key conclusions with information from multiple sources. The Evaluation Team is aware of the methodological challenges associated with evaluating TA, particularly TA for capacity building. Some of the challenges were identified in the Independent Evaluation Office’s evaluation of IMF TA. The Evaluation Team tried to address those challenges by using triangulation but the information base was incomplete and a considerable amount of judgment was applied. The evaluation methodology was designed to make those judgments transparent to readers. 
19.	The Evaluation Team used a number of reports and databases including: (i) material and reports available on the PFTAC webpage; (ii) other unpublished material and reports produced by PFTAC; (iii) reports, documents and data from IMF; (iv) IMF’s International Financial Statistics; (v) IMF’s past evaluations of PFTAC; (vi) the World Bank’s governance indicators; (vii) the World Bank’s Statistical Information Database; and (vi) selected reports available from the websites of AUSAID, NZAID,  ADB and the Pacific Islands Forum Secretariat. 
20.	Given the nature of PFTAC’s work and the range of services that was provided, the Evaluation Team relied, to a considerable extent, on perceptional data, i.e., opinions, views and comments made by various key informants and answers on questionnaires. The Evaluation Team attempted to validate the perceptions of the various parties and compare the perceptions with material on the written record and in various reports. The Evaluation Team applied its best judgment, based on its assessment of the totality of the evidence.
C.	Electronic Survey
21.	The Evaluation Team designed a web-based survey using SurveyMonkey. The names, titles, employers and E-mail addresses were assembled for each person on the master list. This involved considerable time and effort. A dynamic approach was used to develop the master list. Names were constantly added to the list from March to May 2009. PFTAC’s support and that of APD, FAD, MCM and STA in developing a reasonably complete list of the potential survey population is gratefully acknowledged. The following categories of people were included on the master list: 
(i)	TPRC members or people who have attended TPRC meetings as observers;
(ii)	Senior government officials (e. g., Central Bank Governors; Ministers of Finance; secretaries of finance, deputy governors/assistant ministers of finance; heads of statistical agencies; departmental director generals; etc); 
(iii)	Key technical counterparts (e.g., chief accountant, head of treasury, head of banking supervision department, debt manager, etc.) who have the day-to-day experience of working with the Resident Advisors;
(iv)	Participants in PFTAC workshops/training sessions held from 2005 onwards; 
(v)	Staff in APD, FAD, MCM and STA who worked on the countries covered by PFTAC since 2005 and who were familiar with PFTAC’s work. APD, FAD, MCM, STA and OTM provided lists of the concerned staff. 
(vi)	Past and present PFTAC Coordinators and Resident Advisors;
(vii)	Short term experts involved in PFTAC employed by IMF since 2005;
(viii)	Officials and consultants from other TA providers who were knowledgeable of PFTAC operations (i.e., ADB; AUSAID, NZAID and the World Bank); and,
(ix)	Other people met when the Evaluation Team undertook its field work, including staff from the Pacific Islands Forum Secretariat.

22.	Using a web browser, the questionnaire was designed that was patterned on the AFRITAC Evaluation Survey. Questions were formulated using the following considerations: (i) the objectives and background information given in the PFTAC Program Documents; (ii) the relevance, effectiveness, efficiency and sustainability dimensions of the evaluation approach and methodology; and, (iii) the need to assess results for PFTAC as a whole as well as by functional area. To increase response rate and prevent respondents from aborting the survey, the questionnaire was designed to be completed in 10 to 15 minutes. 
23.	The Evaluation Team sent out invitations to participate in the surveys to the target respondents on 6 April 2009. Survey Monkey’s list management tool was used to identify those who had not responded and to send follow-up reminders. During every meeting, the Evaluation Team explained that there would be an electronic survey and requested people to cooperate by completing it. Since the E-mail requesting people to complete the survey came from a non-IMF server, there was some concern that it was not legitimate and might be spam or an attempt at phishing. To address this concern, an E-mail was sent from the PFTAC Coordinator to assure potential respondents that it was a legitimate survey being undertaken for the PFTAC Evaluation. When the first reminder was sent out on 20 April, 145 responses had been received. By the time that the second reminder was sent out on 28 April, 230 responses had been received. On 4 May, the survey results were downloaded for this report a total of 632 questionnaires had been sent out. However, E-mails could not be delivered to 52 E-mail addresses and 11 people declined to participate in the survey. Thus there were 569 questionnaires successfully delivered. From these, 266 responses were received. In addition to downloading all responses, four special runs were undertaken, one each for the groups of respondents who had sufficient knowledge to provided detailed assessments of public financial management, revenue administration, financial sector supervision and statistics.
V.	Reporting

24.	The Evaluation Team produced the following deliverables:
(i)	A detailed work plan was prepared in March 2009 that provided: (i) an overview of how the evaluation was to be conducted; (ii) details of the information collection and analysis methodology, including plans for field visits and meetings; and (iii) information on the roles and responsibilities of the team members. Selected IMF staff at Headquarters and in the PFTAC were given an opportunity to review the evaluation approach and methodology at the beginning of the evaluation. No serious objections to the proposed approach were raised. In practice, the Evaluation Team fine tuned the methodology as the evaluation progressed.
(ii)	A draft evaluation report giving the team’s main findings and recommendations, and supporting analysis was circulated for review and comment by IMF Headquarters Staff and PFTAC was submitted on 8 May 2009. Factual errors were corrected based on the feedback received. Comments received were reflected in the report at the discretion of the Evaluation Team. Major points that were not reflected were included in a footnote, explaining why the Evaluation Team opted not to do so.
(iii)	A revised draft report was submitted for the June 2009 TPRC meeting.
(iv)	The final evaluation report was submitted to OTM in late June 2009 after comments on the draft report from TPRC were considered by the Evaluation Team and the final survey results were downloaded. 

VI.	The Evaluation Team
25.	The Evaluation Team consisted of three independent experts:
(a)	Bruce Murray, team leader and evaluation specialist: Mr. Murray has 35 years of experience, including being the director general of evaluation at the Asian Development Bank. He is currently an independent consultant and an adjunct professor with the Asian Institute of Management. Recently he was the team leader for the evaluation of the three AFRITACs and IMF’s TA program for Iraq, including evaluating the related statistics TAs. For this assignment he was responsible for developing the evaluation approach and methodology, designing the survey instrument and analyzing the results, managing the team, coordinating the preparation of the report and evaluating the statistics TAs.
(b)	Richard Abrams, financial sector supervision expert: Mr. Abrams has 30 years of experience mostly in IMF, where he was responsible to planning, delivering and supervising financial sector TAs. He was also the CARTAC Center Coordinator. Prior to joining IMF, Mr. Abrams was a financial economist with the Federal Reserve Bank of Kansas City. For this evaluation, Mr. Abrams was responsible for evaluating all areas of the support provided by PFTAC in the financial sector supervision area and make inputs to all parts of the report, particularly those portions assessing the PFTAC model. 
(c) 	Kolone Vaai, fiscal expert: Mr. Vaai’s 30 years of professional experience includes holding many senior positions in Samoa including six years as the Secretary of Finance. He has worked on a broad range of assignments as a consultant throughout the Pacific Region since 1991. Mr. Vaai was involved in the establishment of PFTAC and in the 2004 evaluation. Mr. Vaai was responsible for evaluating all areas of PFTAC’s assistance in the fiscal area, including both PFM and revenue administration.