Evaluation of NRC’s Contribution to TRIUMF

Executive Summary

This report presents the results of the 2013 evaluation of NRC's contribution to TRIUMF conducted by an independent evaluation team with the NRC Office of Audit and Evaluation. The evaluation period covers 2008-09 to 2012-13 inclusive. The purpose of the evaluation was to assess the relevance, and the performance of NRC's contribution to TRIUMF.

TRIUMF is a national subatomic physics laboratory conducting research in nuclear and particle physics, nuclear medicine, molecular and materials science, and accelerator science and technology. NRC provides approximately $40 M per year for TRIUMF's core operations, while a number of other organizations, including federal funders, provide funding for capital infrastructure and support TRIUMF researchers and research activities. Overall, the findings of the evaluation of NRC's contribution to TRIUMF show that the activities undertaken by TRIUMF represent good value-for-money for NRC and for Canada. The implementation of the recommendations will be important to enable TRIUMF to continue to perform at a very high level, support the needs of the research community in subatomic physics research and Canada's S&T strategy and yield greater impacts from their commercialization related activities.

The key findings, assessment, recommendations as well as the management responses are summarized below under each of the core issues evaluated (i.e., relevance, performance, resource utilization and governance).

Table 1: Key Findings, Assessment, Recommendations and Management Responses

Key Findings: Relevance	Assessment
Continued Need for the Program
Alignment with the Needs of the Canadian Subatomic Physics Community: There is a growing subatomic physics community in Canada. TRIUMF is aligned with their needs and addresses them in a variety of different ways. TRIUMF also has adequate consultation processes in place to ensure that it participates and invests in equipment and facilities that are relevant to the physics community. Beyond the Canadian subatomic physics community, TRIUMF also meets the needs of a number of other stakeholders, including international researchers, Canadian physicists working in areas other than subatomic physics, Canadian medical research organizations and Canadian industry. Ability to Meet Stakeholder Needs in the Absence of TRIUMF: TRIUMF's equipment and facilities are used to full capacity. There is no duplication of the TRIUMF facility nationally, and TRIUMF is internationally relevant. While TRIUMF has some comparable facilities to those found internationally, the activities are different and the experiments are mostly complementary.	TRIUMF continues to address a demonstrable need in supporting subatomic physics in Canada and TRIUMF plays several distinct albeit interrelated roles in supporting the Canadian and international scientific communities, along with a number of other important stakeholders.
Alignment with Government and NRC Priorities
Alignment with Federal Government Priorities: TRIUMF is well aligned with the 2007 S&T Strategy and the priorities of the Government of Canada. Alignment with NRC Priorities: TRIUMF is aligned with NRC's second strategic outcome: "R&D infrastructure for an innovative and knowledge-based economy."	TRIUMF is well-aligned with federal government priorities overall, and is also aligned with one of NRC's two strategic outcomes.
Role of Government
Appropriateness of Federal Involvement: Large-scale, national endeavours require the participation of the federal government. Appropriateness of NRC Involvement: No alternative funding agent appears to be more appropriate than NRC to oversee the funding to TRIUMF. NRC administers other scientific facilities, and has the organizational competency as well as an understanding of the global science context.	NRC's current role as contributor to TRIUMF is appropriate.
Recommendation 1: NRC should continue to provide stewardship to TRIUMF for the benefit of Canada's research community and in support of the government of Canada's S&T strategy.
Management Response 1: Recommendation Accepted NRC will continue to provide stewardship to TRIUMF by managing TRIUMF's Contribution Agreement and by maintaining the Advisory Committee on TRIUMF (ACOT) which provides advice on scientific and technological issues related to the facility. NRC will continue to participate in an ex officio capacity on TRIUMF's Board of Management and on the Board's Audit Committee to ensure the continued coherence of the governing bodies administering TRIUMF and the responsible management of resources.

 

Key Findings: Performance	Assessment
Scientific Benefits
The International Peer Review of TRIUMF found that TRIUMF's research activities had generated scientific benefits to the Canadian and worldwide communities and met or exceeded the expectations of TRIUMF's plan. While TRIUMF is significantly smaller than many international research facilities, TRIUMF research is of high quality and high impact to the international community. TRIUMF has contributed significantly to the generation of scientific knowledge by its users.	TRIUMF plays a central role in subatomic physics in Canada. It has generated significant scientific knowledge for Canada and the world through research conducted by its researchers, research conducted by users of TRIUMF's facilities and participation in important international collaborations.
Canada's Position in Subatomic Physics
Canada holds a leadership position in many of the fields in which TRIUMF is active, in large part due to TRIUMF's efforts. TRIUMF is central to much of the Canadian effort in subatomic physics.	TRIUMF has greatly enhanced Canada's position among the world's leaders in physics through its support to Canadian researchers and involvement in international projects.
Social and Economic Benefits
Contribution to Economic Benefits: TRIUMF has contributed to the achievement of economic benefits by Canadian firms. There are future economic benefits that will be realized through TRIUMF and AAPS's activities, although improvements to TRIUMF and AAPS's commercialization practices could increase the number of future benefits. TRIUMF's short-term ROI to Canada is estimated to be at least 1.7 dollars for every dollar invested, which is significant considering its fundamental research mandate. Contribution to Social Benefits: TRIUMF's research has led to several health benefits, the most notable being the production of medical isotopes using non-reactor technologies. TRIUMF has contributed significantly to the development of the Canadian subatomic physics community through both their training programs and role in attracting researchers to Canada. TRIUMF's outreach activities are appropriate and have contributed to the elevation of public awareness of TRIUMF and subatomic physics in Canada.	Considering its primary mandate in the area of fundamental research, TRIUMF has made noteworthy contributions to the private sector, the health sector and in supporting the development of the future Canadian subatomic physics community. However, the impact of TRIUMF's commercialization-related activities could be enhanced with the implementation of improved practices in this area.
Recommendation 2: NRC should encourage TRIUMF to be more strategic with its commercialization activities including investigating and adopting best practices.
Management Response 2: Recommendation Accepted NRC and ACOT have expertise and experience that can assist TRIUMF in the area of industrial partnerships and commercialization. NRC will work with TRIUMF on a Business Development Plan which will outline measurable goals and targets for the commercialization of TRIUMF technologies and the adoption of other best practices to improve the overall impact of commercialization activities. A draft of this Plan will be presented to ACOT and the Plan will be finalized as part of entering into Five-Year Plan 2015-2020.

Key Findings: Resource Utilization	Assessment
Economy and Efficiency
TRIUMF is more efficient than other international facilities and programs operating in similar fields. Costs related to NRC's administration of its contribution agreement to TRIUMF are very low. TRIUMF has been able to generate outputs and progress towards expected outcomes, despite the dual challenge of increased input price pressures and static level of operational funding. However, several areas for improvement were identified: TRIUMF does not have a comprehensive performance management system in place; Concerns were also raised regarding the on-going maintenance of the infrastructure of the facility and its ability to continue to sufficiently support operation once ARIEL comes online; Concerns were raised around the lack of critical mass in two of TRIUMF's areas of research.	In comparison to other facilities, NRC's costs to administer the funding agreement and TRIUMF's management of its facilities appear to be more efficient. While available evidence suggested that achieving a sufficient level of outputs and outcomes with a further minimization of resources would have been unlikely, the refinement of TRIUMF's performance management plan may provide additional insights to more strategically manage its resources and find efficiencies. Strain on TRIUMF's operational budget may have hindered its progress to achieve expected outcomes. In addition, several external factors have impacted TRIUMF's cost-effectiveness beyond their control.
Recommendation 3: To support NRC in promoting the value and accountability of TRIUMF, TRIUMF should define and implement an effective performance management system that is aligned with its strategic objectives and operational objectives.
Management Response 3: Recommendation Accepted TRIUMF already has a logic model that lays out the relationships between the resources, activities, outputs and outcomes of its activities. NRC will work with TRIUMF to define key performance indicators that can be tracked over time in a consistent manner in order to measure TRIUMF's progress in key activities. The framework for performance indicators will be discussed with ACOT in the 2014-2015 fiscal year and will be incorporated into TRIUMF's annual reporting process by 2015-2016.
Recommendation 4: NRC should ensure that TRIUMF has an appropriate mechanism in place enabling them to prioritize and manage the ongoing maintenance of infrastructure at the facility.
Management Response 4: Recommendation Accepted NRC will work with TRIUMF to ensure that a Risk Assessment Framework is in place that will assist in identifying maintenance issues and prioritizing reinvestment in the facility. The Risk Assessment Framework will be discussed annually with ACOT.
Recommendation 5: NRC should ensure that TRIUMF has a plan in place to prioritize its financial and human resources to achieve critical mass in its areas of research.
Management Response 5: Recommendation Accepted Working with TRIUMF Management, NRC will ensure effective mechanisms are in place that will enable TRIUMF to prioritize across key initiatives, allowing it to function most effectively within its allocated budget. As the laboratory begins its 2015-2020 performance period, an operating plan reflecting these priorities will be presented to ACOT and ACT.

Key Findings: Governance	Assessment
Governance
Governance Best Practices: TRIUMF's governance model includes many best practices. Opportunities for Improvement to TRIUMF's Governance Model: The Agency Committee on TRIUMF (ACT) has not been used effectively to coordinate TRIUMF's activities. Governance of Other International Physics Facilities: TRIUMF's governance structure has advantages over those used by other international physics facilities such as stable, long term funding.	While TRIUMF's governance structure has many similarities to other comparable international physics facilities, it also has a number of advantages over these governance models. There is an agency coordination committee but it has not been used to its full potential.
Recommendation 6: NRC should convene the Agency Committee on TRIUMF (ACT) to discuss and better coordinate the plans for TRIUMF. Furthermore, the expected role of the ACT should be clarified as well as its membership reviewed through an update of its Terms of Reference and communicated to stakeholders to ensure a consistent understanding of its role.
Management Response 6: Recommendation Accepted A meeting of the Agency Committee on TRIUMF (ACT) will be convened early in FY 2014-15 to discuss TRIUMF's Strategic Plan and the Evaluation of the facility as well as future directions for ACT.

Acronyms and Abbreviations

AAPS	Advanced Applied Physics Solutions Inc
ACOT	Advisory Committee on TRIUMF
ACT	Agency Committee on TRIUMF
AECL	Atomic Energy of Canada Limited
ARIEL	Advanced Rare IsotopE Laboratory
BC	British Columbia
BCKDF	BCKDF British Columbia Knowledge Development Fund
CERN	European Centre for Nuclear Research
CFI	Canada Foundation for Innovation
CLS	Canadian Light Source
CMMS	Centre for Molecular and Materials Science
E-Linac	Superconducting Electron Linear Accelerator
FTE	Full-time equivalent
HQP	Highly qualified personnel
IPRC	International Peer Review Committee
ISAC	Isotope Separator and Accelerator
ITAP	Isotope Technology Acceleration Program
LHC	Large Hadron Collider
MSEs	Methodological and Subject Matter Experts
NRC	National Research Council
NRCan	Natural Resources Canada
NSERC	Natural Sciences and Engineering Research Council
OAE	Office of Audit and Evaluation
PAA	Program Activity Architecture
PDF	Post-doctoral fellow
PSAC	Private-Sector Advisory Committee
RAL	Rutherford Appleton Laboratory
ROI	Return on Investment
S&T	Science and technology
TAI	TRIUMF Accelerators Inc.
Tc-99m	Technetium-99m
UBC	University of British Columbia
UNESCO	United Nations Educational, Scientific, and Cultural Organization
USA	United States of America

1. Introduction

TRIUMF functions as a national laboratory and as Canada's gateway to the international subatomic physics community. Physically located on the south campus of the University of British Columbia (UBC), TRIUMF is owned and operated by a consortium of Canadian universities from across the country. NRC's role is to administer operational funding to TRIUMF and to monitor all matters related to the contribution agreement. The NRC funding provides operating funds to TRIUMF, while other organizations, including other federal funders, provide support that allows TRIUMF to pursue sponsored research and activities.

An evaluation of National Research Council's (NRC) contribution to TRIUMF was required at this time to meet the requirements of the 2009 Treasury Board Policy on Evaluation and the Financial Administration Act, Section 42.1. All ongoing transfer payment programs must be evaluated every five years. TRIUMF has had a number of Peer Reviews which examined the science excellence of the facility. The last such assessment took place in 2008-09 when an international peer review committee (IPRC) assessed TRIUMF's 5-year strategic plan and the degree to which the laboratory delivered on that plan.

The 2008-09 to 2012-13 evaluation of NRC's contribution to TRIUMF was led by an independent evaluation team from the NRC Office of Audit and Evaluation (OAE). The work of the evaluation team was supported by Methodological and Subject Matter Experts (MSEs) who provided advice related to the evaluation framework, approach, instruments, interpretation of findings, and recommendations. These experts played an advisory role in ensuring a high-quality and useful evaluation product. The granting councils were represented on the MSE committee and were invited to attend the IPRC meetings.

Following the evaluation overview presented below, Section 2.0 of this report provides a profile of the program. Sections 3.0 through 6.0 present the evaluation study's findings organized by broad evaluation issue (relevance, performance, efficiency, economy, and governance). Section 7.0 includes the overall conclusions of the evaluation.

1.1 Evaluation Overview

This evaluation of NRC's contribution to TRIUMF was undertaken in fiscal year 2013-14. As the fulfillment of TRIUMF's goals draws upon resources beyond those provided by NRC, the full slate of activities and outcomes of TRIUMF were considered as part of the evaluation.The evaluation assessed the value-for-money of TRIUMF, including relevance, performance, efficiency and economy, and covers the period of 2008-09 to 2012-13.

The approach and level of effort for the evaluation was commensurate with the low level of risk of the program. The specific evaluation questions, outlined at the start of each section, are based on consultations with TRIUMF, the NRC VP office, Emerging Technologies, responsible for the oversight of TRIUMF and a review of key documents during the planning stage. The evaluation issues are also aligned with the requirements of the 2009 Treasury Board Policy on Evaluation. The complete evaluation matrix, which identifies the lines of evidence employed to respond to each question, was developed by the evaluation team in consultation with the MSEs, and is presented in Appendix A.

The evaluation methodology integrated the use of multiple lines of evidence and complementary research methods as a means to enhance the reliability and validity of the information and data collected. The specific methods used in the study include:

• Internal and external document review;
• Administrative and performance data review;
• Key informant interviews (internal interviewees (9); external interviewees (26), and included representatives from funders, collaborators, associations, international and Canadian physics facilities, and full, associate and non-member universities);
• Survey of users (250);
• Comparison study of selected international physics facilities; and
• International peer review (Committee membership may be found in Appendix F).

For key informant interviews, the following scale is used in the text of the report to indicate the relative weight of the responses for each of the respondent groups.

Table 1: Scale of views and opinions of respondents

All/almost all	90% or more
Many/most	at least 50% but less than 90%
Some/several	at least 25% but less than 50%
A few	at least two respondents but less than 25%

Additional studies were commissioned by both TRIUMF and NRC to complement the methods used in the evaluation. TRIUMF commissioned a bibliometric study and the NRC Office of the Vice-President, Emerging Technologies Division commissioned a return on investment (ROI) study, results of which have been incorporated into this evaluation report.

A more detailed description of the study methodology, limitations and challenges is provided in Appendix B.

2. Program Profile

The purpose of the program profile is to provide all readers with an overview of the program and how it has been defined for the evaluation. TRIUMF is a subatomic physics laboratory conducting research in several different research areas, including nuclear and particle physics, nuclear medicine, molecular and materials science, and accelerator science and technology (S&T).^{Footnote 1} These research areas require large-scale, expensive facilities for experimentation. As such, TRIUMF maintains unique scientific infrastructure to be accessed by members of the Canadian and international scientific communities on a quid pro quo basis including a world-class rare isotope beam facility (ISAC), an electron accelerator (E-linac), a Tier-1 Data Centre for the ATLAS experiment at CERN, and a 500 MeV cyclotron. As international researchers access TRIUMF facilities, this provides an avenue by which Canadian researchers may access other international facilities. In addition to its academic pursuits, TRIUMF's outreach activities strive to communicate its activities to those outside the physics community.

TRIUMF, originally known as the Tri-University Meson Facility, began as a university research facility for nuclear physics but has evolved into a national laboratory with a broader range of activities in subatomic physics. TRIUMF was established in 1968 through an agreement among the federal government of Canada and Simon Fraser University, UBC, and the University of Victoria, referred to as the consortium of universities. This consortium has since grown to include 18 Canadian universities, of which eleven are full members and seven are associate members of TRIUMF. The eleven full members share the legal and financial liability for TRIUMF and have voting privileges on the Board of Management. Each full member university shares an equal interest in all assets except for the land and buildings, which are owned by UBC as the laboratory is located on UBC land. While there is no formal cost to be a full or associate member, their commitments include steering their on-campus research activities and resources in directions supportive to TRIUMF, providing salary resources to support half of a joint faculty position with TRIUMF, and overseeing and managing TRIUMF-related Canada Foundation for Innovation (CFI) projects. The responsibility for the federal contribution to TRIUMF was transferred from the Atomic Energy Control Board to NRC in 1976.

NRC's role is to administer operational funding to TRIUMF and to monitor all matters related to the contribution agreement. It also plays a role in overseeing TRIUMF's activities (e.g., as a non-voting member of the TRIUMF Board of Management and the TRIUMF Audit Committee and monitoring TRIUMF's funding through its Corporate Finance Branch). Additional oversight of TRIUMF operations is also conducted by the Advisory Committee on TRIUMF (ACOT) and the Agency Committee on TRIUMF (ACT).

Table 2: Member and Associate Universities

Members	Associate Members
University of British Columbia	University of Calgary
Simon Fraser University	University of Toronto
University of Victoria	York University
University of Alberta	McGill University
Carleton University	McMaster University
University of Guelph	University of Northern British Columbia
University of Manitoba	University of Regina
Université de Montréal	Saint Mary's University
Queen's University	University of Winnipeg

Source: TRIUMF website

2.1 Program Goal, Objectives, and Activities

TRIUMF's over-arching goal is to be a world leader in subatomic physics research. The main objectives for TRIUMF over the evaluation period were to:

• Act as Canada's connection with international particle physics research facilities so that Canadian scientists can become involved in research programs at the forefront of particle physics and related sciences;
• Advance science and support Canada's subatomic physics community by operating a world class facility that maintains innovative research programs in nuclear physics, particle physics, accelerator S&T, materials sciences and nuclear medicine;
• Maximize the economic and societal benefits of the federal government's investment in TRIUMF through technology transfer and support to Canadian companies and non-profit organizations.

Figure 1: TRIUMF's Main Research Areas, Activities, Competencies, and Outcomes

2.2 Key Stakeholders

Key stakeholders of TRIUMF's efforts include Canadian and international researchers, university students and post-doctoral fellows; Canadian medical research organizations; Canadian firms involved in nuclear medicine and other isotope-science applications; Canadian high-technology small and medium-sized enterprises; and Canadian high school students and teachers.

2.3 Program Resources

This section presents TRIUMF's financial and human resources from 2008-09 to 2012-13. Over the evaluation period TRIUMF was successful in attracting over $360 M in funding and revenue. A summary illustration of how TRIUMF manages these funds, as well as the relative size of each category, is provided immediately below (Figure 2). Specific details about expenditures are presented in Figure 3 and more exhaustive details about funding and revenue sources are presented in Section 5.0: Resource Utilization.

Figure 2: Summary of TRIUMF Funding, Revenue and Designated Expenditure Areas (2008-09 – 2012-13)

Long description of TRIUMF Funding, Revenue and Designated Expenditure Areas

Source of revenue or funding			Designated expenditure areas			Total (2008-09 - 2012-13)
NRC Contribution Agreements			Core Operations Staffing Maintainence and repairs Basic infastructure Utilities			$220.5 million ($96.4 million of which is A-based)
Competitive Funds and Collaboration Agreements			Sponsored research Sponsored activities (E.g. NSERC experiments, CFI capital projects, cost recovery, Tc-99m project)			$131.6 million
Commercial Revenues and Interest			Enhancements New initiatives Contingency			$9.8 million

TRIUMF's expected outcomes:

Source: TRIUMF financial database

As the illustration above shows, the funding that TRIUMF receives from the NRC contribution agreement^{Footnote 3} is used for 'core operations'. This provides the financial foundation for the fulfillment of TRIUMF's mandate. In addition, supplementary revenue is received from competitive funds (e.g., CFI,^{Footnote 4} Natural Sciences and Engineering Research Council (NSERC), British Columbia Knowledge Development Fund (BCKDF), and Natural Resources Canada (NRCan)), collaboration agreements (e.g., with Nordion, AAPS, and BC Cancer Agency), and, to a lesser degree, commercial and investment revenue sources. The activities that this supplementary revenue allows TRIUMF to pursue include 'sponsored research' and 'sponsored activities' - e.g., NRCan's Isotope Technology Acceleration Program (ITAP) contract to TRIUMF for finding alternative (non-nuclear) sources of a key isotope (technetium-99m) – and supplementary facility enhancements, initiatives, and contingency.

2.3.1 TRIUMF Expenditures

TRIUMF's annual average expenditures were $71.7 M over the course of the evaluation period. Since 2008-09, overall expenditures have increased by 39.1% (see Figure 3). This increase is largely attributed to the building and equipment costs associated with the construction of Advanced Rare IsotopE Laboratory (ARIEL) and building improvements. TRIUMF has been successful in leveraging NRC funding and securing additional short term funding from several sources for these investments. Salaries and benefits have remained relatively constant in nominal terms over that same period.

Figure 3: TRIUMF Expenditures by Type

Long description of TRIUMF Expenditures by Type

Expenditures	2008-09	2009-10	2010-11	2011-12	2012-13
Travel	$1,920,396	$1,833,428	$1,847,246	$1,949,168	$1,778,402
Telecommunications	$158,169	$245,542	$190,173	$216,387	$142,329
Supplies and Other Expenses	$13,763,186	$11,527,208	$10,720,245	$13,355,698	$12,817,742
Salaries and Benefits	$36,638,698	$40,102,553	$40,281,034	$40,588,275	$40,434,297
Power	$2,343,671	$2,551,233	$2,608,866	$2,724,992	$3,092,094
Facility Conformity Costs	$1,000,000	$1,000,000
Equipment	$4,116,790	$4,342,100	$5,251,722	$6,662,677	$10,097,681
Consulting	$1,348,735	$1,347,242	$2,390,710	$3,374,520	$1,759,592
Computer	$903,876	$1,203,207	$2,049,025	$1,241,276	$1,277,465
Building and Improvements	$1,176,123	$334,315	$847,283	$6,387,794	$16,750,364

Source: TRIUMF financial database

2.3.2 Human Resources

On average, TRIUMF employed 385 full-time equivalents (FTEs) annually between 2008-09 and
2012-13, of which 328 FTEs were funded by NRC.^{Footnote 5} In addition, TRIUMF supported an average of 44 post-doctoral fellows (PDFs), 39 graduate students and 76 unique undergraduate students each year. These numbers remained relatively consistent over the evaluation period.

Figure 4: TRIUMF Human Resource Data

Long description of TRIUMF Human Resource Data

Human Resource	2008-09	2009-10	2010-11	2011-12	2012-13
Post-Doctoral Fellows	39	43	43	45	49
Administrative	44	47	50	42	45
Computing	26	28	27	27	31
Director	1	1	1	1	1
Engineering	68	74	75	72	77
Faculty Scientist	2	1	1	1	0
Research Scientist	55	49	54	55	47
Technical	179	175	175	165	174

Source: TRIUMF human resources database

3. Findings: Relevance

The relevance of TRIUMF was assessed by examining the current and future needs of TRIUMF's stakeholders as well as the extent to which TRIUMF activities are aligned to these needs. Additionally, the activities of TRIUMF were assessed according to their consistency with federal roles and responsibilities and their alignment with federal government priorities and NRC's current strategic outcomes.

3.1 Continued Need for the Program

3.1.1 Alignment with the Needs of the Canadian Subatomic Physics Community

Key Finding 1: There is a growing subatomic physics community in Canada. TRIUMF is aligned with their needs and addresses them in a variety of different ways. TRIUMF also has adequate consultation processes in place to ensure that it participates and invests in equipment and facilities that are relevant to the physics community.

The International Peer Review Committee (IPRC) affirmed that TRIUMF plays an important role in supporting the Canadian subatomic physics community and has been successful in supporting this role in a number of ways. For instance in the area of the particle physics, the IPRC concluded:

The evaluation found that the subatomic physics community in Canada has grown in the past five years. According to the 2011-2016 NSERC subatomic long range plan, "the number of subatomic physicists in Canada continues to grow, with an average of six new faculty hires per year in the last five years."^{Footnote 6} This trend is in line with the growing membership of the Canadian Association of Physicists (CAP), a broader association of Canadian physicists with two divisions aligned with TRIUMF's primary research areas: nuclear and particle physics. The Division of Nuclear Physics currently has 212 members and the Division of Particle Physics has 288 members. Both of these areas have shown a growth in membership (34% and 22%, respectively) since 2009.

To align its activities with the future needs of the Canadian subatomic physics community, TRIUMF engages with the community to identify their priority areas and determine where TRIUMF could play a role. TRIUMF employs various mechanisms including extensive consultations for the development of TRIUMF's five-year strategic plan. Survey responses showed that TRIUMF was successful in aligning its equipment, facilities and expertise with the needs of the community. Almost all of TRIUMF users surveyed as part of the evaluation indicated that TRIUMF invests in equipment and facilities that are currently relevant to their research community (96%) and that access to TRIUMF's equipment and facilities resulted in data that met their research needs (92%). The relevance of TRIUMF's facilities to the physics community is further evidenced through the subscription rate of TRIUMF facilities. Subscription data was available for several TRIUMF facilities (ISAC-I/II, μSR, β-NMR), all of which were oversubscribed for the evaluation period. Specific subscription information was not available for all TRIUMF facilities. However, TRIUMF's machine shop is also oversubscribed with an average backlog of 1-2 months. TRIUMF management, as well as the IPRC, have indicated that some of TRIUMF's facilities are aging and need maintenance and/or replacement. The impact is discussed in Section 5.0: Resource Utilization.

Beyond the facilities available at TRIUMF itself, the importance of TRIUMF's participation, and leadership, in globally relevant, cutting-edge experiments was raised as an important way in which TRIUMF supports the scientific community. TRIUMF's participation in key international experiments enables Canadian scientists to participate in those, as well as other, important research projects. According to 92% of survey respondents, TRIUMF is a hub for international science. In order to act as a gateway to prestigious international projects, such as ATLAS and T2K, and ensure that these projects maximize their chances of significant discoveries, TRIUMF welcomes and attracts international users. Currently TRIUMF supports a community of 500 users who are affiliated with foreign institutions. These users represent roughly 65% of TRIUMF's total user community. In return for their access to TRIUMF's facilities, Canadian researchers are welcomed in foreign physics facilities.

TRIUMF has also met stakeholder needs in terms of training HQP and contributing to the ongoing development of the Canadian subatomic physics community. For example, 76% of surveyed TRIUMF users agreed or strongly agreed that access to TRIUMF contributed greatly to their education and training. Furthermore, 90% of the users surveyed agreed or strongly agreed that TRIUMF was critical to the training of HQP. A few interviewees also noted that TRIUMF is important to student training, since it provides students with hands-on experience.

Key Finding 2: Beyond the Canadian subatomic physics community, TRIUMF also meets the needs of a number of other stakeholders, including international researchers, Canadian physicists working in areas other than subatomic physics, Canadian medical research organizations and Canadian industry.

In addition to supporting segments of the Canadian and international physics communities, TRIUMF has been able to help meet the needs of other stakeholders groups, including Canadian industry and a number of Canadian medical research organizations. TRIUMF provides access to facilities, technical services and research for industry (e.g., Nordion, PAVAC, IKOMED) that is not available elsewhere in Canada. TRIUMF's recent advancements in nuclear medicine research have been developed with a small community of medical research organizations (e.g., BC Cancer Agency, Pacific Parkinson's Research Centre) to target health issues of importance to Canada and internationally. TRIUMF's research in alternative production methods for medical isotopes (specifically technetium-99m) has responded to a Canadian, and worldwide, need that was not expected at the time of TRIUMF's last five-year plan; however, the facilities and expertise residing at TRIUMF enabled it to respond to this unanticipated need. While not part of TRIUMF's original activities, the nuclear medicine program has been expanded to become one of TRIUMF's main research programs and formalizes the Canadian medical community as one of the numerous stakeholders TRIUMF supports.

3.1.2 Ability to Meet Stakeholder Needs in the Absence of TRIUMF

Key Finding 3: TRIUMF's equipment and facilities are used to full capacity. There is no duplication of the TRIUMF facility nationally, and TRIUMF is internationally relevant. While TRIUMF has some comparable facilities to those found internationally, the activities are different and the experiments are mostly complementary.

All internal and external interviewees confirmed that there is no duplication of TRIUMF facilities at the domestic level. The Government of Canada does invest in other subatomic physics facilities: SNOLAB and the Perimeter Institute; however the facilities use different approaches meaning they each support different types of experiments. SNOLAB uses "underground approaches" and the Perimeter Institute uses "theory-based approaches", while TRIUMF mainly uses 'accelerator approaches'. The main differences between TRIUMF and the Canadian Light Source are that they pursue different scientific questions, employ different types of accelerators (cyclotron and synchrotron respectively), and TRIUMF's mandate pushes accelerator technology itself while CLS simply applies it for novel research. Given the limited resources in Canada for subatomic physics, all these research facilities collaborate with one another. TRIUMF is a critical contributor to SNOLAB experiments and was also a contributor to the laboratory's initial establishment. Ongoing collaborations include access to facilities, expertise and combined efforts (e.g., the TRISEP summer program for PDFs and PhD students). TRIUMF also acts as a key focal point for the Canadian community, including SNOLAB and the Perimeter Institute, to reach out to the international arena.

Although it is possible for the Canadian physics community to gain access to international physics facilities, only 21% of user survey respondents indicated that it would be "likely" or "very likely" that they could access the appropriate research facilities in the absence of TRIUMF. In addition, 93% of this subgroup responded affirmatively that their research would have to occur outside of Canada. When asked about the main advantage of using TRIUMF, the most frequent response (cited by 53% of those who responded) was "specialty or unique equipment."

"TRIUMF is the only place in Canada that has the comprehensive technical expertise in accelerator physics and engineering that enable Canadians to participate in international scale projects."

– External Interviews, 2013

Interviewees also confirmed that there may be comparable facilities internationally but that the research activities conducted are different and the experiments are mostly complementary to those carried out at TRIUMF, as opposed to being duplicative (i.e., findings from one facility may inform experiments at another, and the equipment may contain slight variations in design that render slightly different experiments possible). TRIUMF also has unique capabilities; for example, one interviewee referenced TRIUMF's unique program in β-NMR, which uses a unique combination of beam particle type and associated instruments to probe materials structure at a level of detection that is orders of magnitude more sensitive than conventional means. Finally, some internal interviewees noted that the existence of TRIUMF in Canada helps to ensure that Canadian research can be conducted in a timely manner.

While research by Canadian scientists may be conducted elsewhere in the world, all interviewees confirmed that this access to other international physics facilities would be compromised in the absence of TRIUMF. Key informants indicated that countries wishing to participate in large collaborative projects, such as in the field of physics, must be able and willing to invest resources and expertise in the research in order to be a partner.

Without TRIUMF, the domestic training of HQP and retention of HQP in subatomic physics research would be limited and may also lead to "brain drain" as scientists conducting research in this discipline may seek opportunities abroad. This concern is supported by UNESCO's most recent world science report (2010) which warns about "the potential loss of Canada's brain power […] and on Canada's future global competitiveness" if Canada does not react to the increase in S&T investment by both the US and other attractive countries ^{Footnote 7}. In all cases, in the absence of TRIUMF, Canada would not be a leader in the field of subatomic physics.

3.2 Alignment with Government and NRC Priorities

3.2.1 Alignment with Federal Government Priorities

Key Finding 4: TRIUMF is well aligned with the 2007 S&T Strategy and the priorities of the Government of Canada.

Evaluative evidence gathered through the document review confirms that TRIUMF's activities are well aligned with federal government priorities outlined in several priority-setting documents. All internal and external key informants agreed that TRIUMF is well aligned with federal government priorities.

TRIUMF's activities align with all three of the advantages from the main S&T strategy document for the Government of Canada, Mobilizing Science and Technology to Canada's Advantage (2007). TRIUMF's activities are best aligned with the Knowledge and People Advantages, although its recent partnerships with other levels of government and the private sector also support the Entrepreneurial Advantage. In summary:

• Knowledge Advantage: TRIUMF is well-aligned because it "ensures that higher education institutions have the leading-edge research equipment and facilities required to compete with the best in the world".^{Footnote 8} TRIUMF's work in nuclear medicine, information technology (e.g., Tier-1 Data Center), and their development of muon geotomagraphy with AAPS, each respectively align with three of the four strategic areas identified under the Knowledge Advantage;
• People Advantage: TRIUMF "helps increase the supply of the highly qualified and globally connected S&T graduates that businesses need to succeed in today's economy." TRIUMF achieves this by: providing access and financial support to students; training professionals; and encouraging the pursuit of S&T careers through outreach activities; and
• Entrepreneurial Advantage: the Government of Canada recommends that institutions like TRIUMF seek opportunities for collaboration with other research institutes, governments, and private sector partners in areas where Canada has the "potential to achieve world-leading excellence." TRIUMF, in their pursuit of world-class research goals, has been successful in attracting collaboration and partnership from a broad and diverse range of sources, including international research institutes (e.g., CERN), domestic research facilities (e.g., SNOLAB and new member universities), private sector partners (e.g., Nordion and PAVAC), and other governments (e.g., funding from the province of BC and other international governments).

In addition to the 2007 S&T Strategy, TRIUMF has also shown alignment with federal government priorities through recent funding initiatives announced in the 2010 and 2012 budgets. Both of these budgets provided additional funding which was awarded through a competitive process to a project team including TRIUMF specifically to pursue solutions to an anticipated global shortage in medical isotopes.

3.2.2 Alignment with NRC Priorities

Key Finding 5: TRIUMF is aligned with NRC's second strategic outcome: "R&D infrastructure for an innovative and knowledge-based economy."

NRC's most recent strategy, announced in 2013-14, has been described as representing a shift in direction for the organization. In implementing this strategy, NRC is focused on market-driven innovation outcomes and benefit to Canada. The Management of National Science and Technology Infrastructure is one of four business lines, which have been put in place by NRC as part of its transformation. TRIUMF is also represented in NRC's revised Program Activity Architecture (PAA), where it is a component of the Science Infrastructure and Measurement program activity area. This program area is aligned with one of NRC's two strategic outcomes (i.e., "R&D infrastructure for an innovative and knowledge-based economy").

3.3 Role of Government

This section explores the role of the federal government in funding TRIUMF, as well as NRC's role on behalf of the federal government.

3.3.1 Appropriateness of Federal Involvement

Key Finding 6: Large-scale, national endeavors require the participation of the federal government.

Historically in Canada, the funding for large scientific research facilities has been a shared responsibility of the federal and provincial governments. Internationally, research infrastructure similar to TRIUMF is housed at national facilities and more than 90% of their funding is derived from federal sources. Examples of foreign laboratories receiving mainly federal funding include the GSI lab in Germany and all of the Department of Energy national laboratories in the USA. In essence, large-scale, national endeavours, such as TRIUMF, require the participation of the federal government.

As noted by several key internal and external informants, this is particularly apparent in the field of fundamental science where private sector companies have little interest in funding basic research. Furthermore, as many stakeholders, both internal and external to NRC pointed out, TRIUMF would not exist without federal funding, as the cost to build the infrastructure and operate the facilities for the long-term would be prohibitive for any individual province. Many key informants from TRIUMF and the physics community highlighted that individual universities do not currently represent a viable alternative to NRC because of their provincial focus and lack of national mandate. Moreover, the high financial commitment needed to build and operate TRIUMF as well as the associated requirement to employ highly qualified personnel to maintain and operate the infrastructure contributes to the creation of a financial obligation that is too large for the consortium of TRIUMF member universities to assume.

Interviewees felt that, in the absence of TRIUMF, Canadian scientists would likely pursue their research in other countries and any economic benefits derived from their work (e.g., the creation of licenses, patents or companies) would be lost. Economic and social benefits will be further described in Section 4.0: Performance of this report.

3.3.2 Appropriateness of NRC Involvement

Key Finding 7: No alternative funding agent appears to be more appropriate than NRC to monitor and transfer funding to TRIUMF. NRC administers other scientific facilities, and has the organizational competency as well as an understanding of the global science context.

As previously indicated, NRC's role is to administer federal funding to TRIUMF and to monitor all matters concerning the management of the contribution agreement. It also plays a role in overseeing TRIUMF's activities.

Many stakeholders, both internal and external to NRC, commented that NRC's role with respect to TRIUMF is appropriate, given the reasons for federal support as well as the facts that NRC is a federal government organization that operates in the national interest and that TRIUMF maintains a national mandate as Canada's national nuclear and particle physics laboratory. Many key informants, both from TRIUMF and from the external Canadian physics community, noted that NRC has both the appropriate knowledge (i.e. understanding of TRIUMF's programs and the global context of science) and the organizational competencies that are required to effectively support TRIUMF. The vast majority of interviewees also considered that NRC's funding model works well and that, in its absence, this administrative structure would have to be duplicated in another organisation.

Some alternative delivery agents for TRIUMF were identified by internal and external key interviewees. However, they did not consider that these alternatives were more appropriate than NRC. The most commonly identified alternatives were CFI and NSERC. A few external stakeholders also advocated for a national scientific body to oversee and fund all large scale scientific infrastructure in Canada, although one does not exist at this time. Some documentation reviewed showed that this has been proposed in the past; however, it has never been implemented.

Evidence collected from the document review, as well as from interviews with internal and external stakeholders, indicated that while CFI does provide funding for large scale infrastructure, its funding is contingent on the universities finding matching funding from other partners such as the provinces. This was raised as a concern by a few internal and external interviewees since provincial priorities may not be the same in all cases. They noted that this requirement would be particularly challenging for national facilities such as TRIUMF. With respect to NSERC, stakeholders consulted as a part of the 2011-2016 Long Range Plan for Subatomic Research felt that NSERC is "ill-suited to support the operations and maintenance costs of a national laboratory such as TRIUMF" because its envelope of funding is exhausted by other research activities.

Overall, all interviewees felt that, given the available options, NRC is best suited to administer the federal contribution to TRIUMF. Some external and internal stakeholders did indicate, however, that opportunities exist for increased collaboration, communication and coordination among all of the primary federal organizations that fund TRIUMF and activities at TRIUMF: NRC, NSERC and CFI.

4. Findings: Performance

As a first step in this evaluation a logic model was developed with TRIUMF's management to serve as the program's road map. It outlines the expected results (i.e., outcomes) of the program, the activities the program undertakes, and the outputs it creates in order to achieve its outcomes. The performance of TRIUMF was examined according to its achievement of expected outcomes, as well as its demonstrated efficiency and economy. The immediate, intermediate and ultimate outcomes associated with TRIUMF's performance can be grouped under three broad interrelated themes. ^{Footnote 9}

• Scientific benefits to Canada and worldwide communities;
• International recognition and participation; and
• Socio-economic impacts.

Each of these themes corresponds to an evaluation question that is addressed in the sections that follow.

4.1 Scientific Benefits

This section of the report examines the contributions of TRIUMF in achieving scientific benefits for Canada and worldwide communities. This includes both research conducted by TRIUMF researchers, as well as research conducted using TRIUMF facilities, supported by TRIUMF personnel.

Key Finding 8: The International Peer Review of TRIUMF found that TRIUMF's research activities had generated scientific benefits to the Canadian and worldwide communities during and met or exceeded the expectations of TRIUMF's plan.

As part of the evaluation the IPRC was asked to assess TRIUMF's past performance as well as provide input into the Strategic Plan developed by TRIUMF for their next five years of activity. The IPRC was made up of nine Canadian and international experts, with expertise in each of TRIUMF's main research areas providing for an informed expert judgment of TRIUMF's performance. The IPRC found that TRIUMF's activities over the past five years were world-class, both when considered as a set, as well as within each of the five research and technology areas reviewed. The following paragraphs highlight the IPRC's findings on the scientific performance of TRIUMF in each area.

Accelerator Science

Accelerators at TRIUMF are both the enabling technology for TRIUMF's research program and a science and technology area in themselves. The IPRC noted that both the applications as well as TRIUMF's work to advance accelerator science and technology are leading edge. As an enabling technology, the IPRC found that "with the recently improved reliability and higher intensities of beams available at ISAC, many of the beams for both unaccelerated and reaccelerated radioactive beams are the best in the world and are attracting important international users." The IPRC report goes on to say that "the recently commissioned superconducting linac of ISAC II is unique in the world." The IPRC also highlighted the key knowledge developed by TRIUMF about accelerators and their associated technologies. Some of the examples highlighted included:

• Development of uranium targets operating at 10 microamps of driver beam to deliver both neutron-rich fission fragments and key heavy elements such as francium and radon for fundamental research;^{Footnote 10}
• Development and implementation of a general purpose element-selective laser resonance ionization source and an ion-guide laser ion source;
• Demonstration of a new production method for the very important medical isotope ^99mTc using small cyclotrons without highly enriched uranium targets;
• Determination of a fundamental limit of intensity in cyclotrons owing to space charge effects; and
• Development of a charge-breeding capability that extends the mass range of ISAC reaccelerated beams to heavy isotopes such as ⁹⁴Sr.

Nuclear Physics

The nuclear physics program at TRIUMF includes nuclear structure, nuclear astrophysics and fundamental symmetries. The IPRC concluded that TRIUMF has achieved significant scientific benefits and generated new knowledge in each of these subprograms and is able to undertake internationally unique experiments in each, due in part of the unique research equipment in place at TRIUMF. The IPRC highlighted some of the important research underway in TRIUMF's nuclear physics program:

• Ground state masses and radii of exotic short-lived halo nuclei such as ¹¹Li and ⁸He;
• Mass measurements of n-rich Ca isotopes, which are also revealing the importance of the three-nucleon force in heavier nuclei;
• Reaction rate measurements using ISAC-I beams have contributed the majority of the world's data for understanding the energy production and nucleosynthesis in stellar media such as Novae and X-ray bursters; and
• Key experiments on superallowed Fermi beta decays (e.g., masses, decay half-lives, branching ratios) for testing the unitarity of the CKM matrix of the Standard Model.

Nuclear Medicine

The IPRC determined that the nuclear medicine program at TRIUMF is world class, even though it only became a full research program at TRIUMF in the past 5 years. The past few years have included some capacity-building activities (e.g., upgrading the nuclear medicine laboratories); however, the IPRC noted that it has already begun to produce scientific outputs, including 98 publications since 2008. The IPRC considered that there exist opportunities for expansion and growth in this area, as there is valuable work underway. However, the committee also assessed that capitalizing on these opportunities would require additional personnel. Some research highlighted by the IPRC included:

• The production of ^99mTc via low energy cyclotrons is a cutting edge technique; and
• The use of nuclear imaging techniques (PET and combined MRI/PET) with ¹⁸F- and ¹¹C-labeled radiotracers for the investigation and treatment of neurological disorders.

Particle Physics

The IPRC confirmed that TRIUMF is involved in some of the most significant particle physics research in the world. Their role in enabling Canada to participate in these experiments is discussed in more detail in the following section. The IPRC highlighted that the "ambitious program in electric dipole moment measurement under the Ultra-Cold Neutron (UCN) program is the leading in-house particle physics activity in TRIUMF." The majority of the work in particle physics is collaborative; much of TRIUMF's work in this area contributes to benefits achieved by a larger group, including the ATLAS, ALPHA and T2K experiments. As described in the IPRC report:

Materials Science

The IPRC acknowledged that TRIUMF's material science program has "an extended record of research excellence and a very good publication record with many in high impact journals." They found that the Centre for Molecular and Materials Science (CMMS) has a diverse user community and they have been able to use the unique instrumentation and sample environment to conduct innovative research. The TRIUMF µSR program in magnetism and superconductivity has addressed many high profile topics such as magnetic order in the pnictide superconductors, orbital currents in cuprate superconductors, magnetic phase diagram of topological superconductors, magnetic charges in spin ice and quantum spin ice, and ferromagnetic semiconductors. TRIUMF scientists have an international reputation for technical innovation in both instrumentation and sample environments. Some notable examples include:

• M20 is the only beamline in the world that combines full spin rotation and high fields with a long time window;
• "Muons on Request" (MORE): extracting only one muon at a time out of a continuous beam using a fast-switching kicker leads to an ultra-low background allowing measurements to much longer times;
• New spectrometers being designed with improved time resolution based on Si PMT detectors;
• 7T replacement for HiTime with improved field homogeneity;
• An oven being designed for high field (9T) spectrometer for measurements up to 600K; and
• The use of new isotopes for β-NMR (¹¹Be, ³¹Mg).

Key Finding 9: While TRIUMF is significantly smaller than many international research facilities, TRIUMF research is of high quality and high impact to the international community.

A bibliometric study was commissioned to examine both TRIUMF and Canada's scientific output in this field. Within Canada, TRIUMF is the third-ranked organization for subatomic sciences scientific output, with almost 2,300 papers included in the data set.^{Footnote 11} As well, TRIUMF's scientific impact (based on the calculated average of relative citations [ARC]^{Footnote 12}) and research quality (based on the average of relative impact factors [ARIF]^{Footnote 13}) approximates their respective national averages.

The study also examined the change in TRIUMF's performance over time (comparing 1996-2000, 2001-2005 and 2006-2010) and found that TRIUMF's output in most fields increased over this period, as did the scientific impact of TRIUMF publications. This means that TRIUMF's more recent papers have a greater impact on the scientific community than their work produced from 1996-2000. From 2001-2010, the observed research quality for subatomic sciences was mostly stable. However, this period does not account for major accomplishments that occurred in the past three years (including many of the major accomplishments highlighted by the IPRC above). The detailed results for each of TRIUMF's research areas are presented in the table below.

Table 3: Change in TRIUMF's Bibliometric Output

Field	Scientific Output			Scientific Impact (ARC)			Research Quality (ARIF)
	1996-2000	2006-2010	Δ	1996-2000	2006-2010	Δ	2001-2005	2006-2010	Δ
Particle Physics	512	536	=	0.99	1.65	↑	1.25	1.24	=
Nuclear Physics	372	430	↑	1.23	1.90	↑	1.23	1.28	=
Accelerator Physics	133	206	↑	1.02	2.22	↑	1.08	1.11	=
Materials Science	46	140	↑	1.29	1.07	↓	0.95	0.91	=
Nuclear Medicine	69	88	↑	0.80	1.84	↑	1.22	1.41	↑
Subatomic Sciences	704	873	↑	1.08	1.56	↑	1.21	1.24	=

Note: "=" denotes equivalence within a range of +/- 15%

As part of the evaluation, TRIUMF was compared against a set of other international physics facilities, ^{Footnote 14} including the bibliometric data for each one. It should be noted that all of these facilities are significantly larger than TRIUMF. The average annual revenue of 7 (of the 9) comparable facilities is almost 4 times larger than TRIUMF's revenue for 2011-12. Not surprisingly, TRIUMF is producing fewer publications than most of the facilities. Nevertheless, the bibliometric data revealed that these smaller sets of publications are of high quality and impact, both within the overall field of 'subatomic sciences' as well as most of the sub-fields. The following table illustrates TRIUMF's performance against the other facilities in terms of bibliometric results.

Table 4: TRIUMF's Rankings among Comparable Facilities (out of 10)

Field	Scientific Output	Scientific Impact (ARC)	Research Quality (ARIF)
Physique des particules	6	4	4
Nuclear Physics	4	8	4
Accelerator Physics	7	2	3
Materials Science	5	6	7
Nuclear Medicine	6	6	2
Subatomic Sciences	6	4	3

This high value of scientific research was echoed through interviews as all interviewees commented on the high value and the significant scientific contributions of TRIUMF, both nationally and internationally. Many internal and external interviewees noted that TRIUMF's researchers have participated in many prestigious international experiments. Three of these experiments have made Physics World's annual top ten list of global breakthroughs in physics, listed below.

• 2012 - Confirmation of the existence of the Higgs boson particle;
• 2011 - Confirmation that muon flavour neutrinos can oscillate into electron flavour; and
• 2010 - Successes in trapping anti-hydrogen.

Finally, the high quality of TRIUMF research is also supported by the national and international recognition of its researchers. Some recent prestigious awards given to TRIUMF researchers include:

• ALPHA-Canada members ^{Footnote 15} - 2014 NSERC John C. Polanyi Award;
• Dr. Pierre Savard - 2013 CAP-TRIUMF Erich Vogt Medal;
• Dr. Jens Dilling- 2012 Radio-Canada Scientist of the Year;
• Dr. Tom Ruth - 2011 Michael J. Welch Award of the Radiopharmaceutical Sciences Council;
• Dr. Alexandre Blais - 2011 CAP Herzberg Medal; and
• Dr. Jess Brewer - 2008 CAP/DCMMP Brockhouse Medal.

Michael J. Welch Award

This highly prestigious award is given by the Radiopharmaceutical Sciences Council to recognize individuals who have made a significant contribution to the field of radiopharmaceutical sciences. In 2011, Tom Ruth, a researcher at TRIUMF, was awarded the prize for his lifelong efforts of applying nuclear science to medicine.

Key Finding 10: TRIUMF has contributed significantly to the generation of scientific knowledge by its users.

Along with the strong scientific achievement from their staff and in-house researchers, TRIUMF's facilities have also been critically important to the achievements of their large group of users (~800). For example, the IPRC noted that several innovative studies by TRIUMF's CMMS user group relied on TRIUMF's infrastructure. Some of these studies have resulted in awards and recognition, such as a Brockhouse Medal ^{Footnote 16} and the election of TRIUMF users to the position of fellow at the American Physical Society.

Users were also surveyed as part of the evaluation to determine the extent to which TRIUMF contributed to their ability to achieve their research outputs. The following table shows that TRIUMF played an important role in the generation of scientific knowledge by its users in all of the listed categories of scientific outputs (conference presentations, invited talks, publications in peer-reviewed journals, and national or international awards or recognition).

Table 5: Role of TRIUMF in Users' Outputs (2008-2013)

User Output	# and % of users who reported output		Role TRIUMF played in attaining output
Minimal	Moderate	Considerable	Essential
Conference presentations	175	87%	1%	14%	23%	62%
Invited talks	140	69%	1%	16%	21%	61%
Publications in peer-reviewed journals	174	86%	2%	6%	28%	64%
National or international awards or recognition	51	25%	2%	18%	35%	45%

Notes: 'No answer' and 'not applicable' groups have been removed. Total population (n) equals 202.

Source: 2013 TRIUMF User Community Survey

4.2 Canada's Position in Subatomic Physics

This section of the report examines TRIUMF's contribution towards Canada's position in physics. This includes both a discussion of Canada's international position in relevant areas of physics and a description of the Canadian landscape, including TRIUMF, in this area.

Key Finding 11: Canada holds a leadership position in many of the fields in which TRIUMF is active, in large part due to TRIUMF's efforts.

The IPRC report states that "Canada plays a significant leadership role in many of the fields in which TRIUMF is active. Given the limited number of Canadian contributors in these fields, Canada's leadership is due in large part to TRIUMF's role. Internationally, TRIUMF is seen as a well-known and well-respected brand, intrinsically linked with Canada's performance in its research areas." Many of the facilities were identified as unique in North America or worldwide and contribute to Canada's leadership in the field. Examples identified include:

• TRIUMF's Accelerator Division has valuable capabilities which are unique internationally;
• TRIUMF's instrumentation related to the generation of intense beams of rare radioactive ions has nearly no comparison worldwide;
• The unique innovation and sample environment available for materials science users;
• µSR is only available at three other facilities in the world and the µSR instrumentation at TRIUMF has some capabilities not available elsewhere; and
• The β-NMR capability at TRIUMF is effectively unique.

TRIUMF is frequently identified by international and Canadian experts as being a piece of scientific infrastructure that is an "advantage for Canada" in the area of Physics and Astronomy

- Report of the Council of Canadian Academies

The IPRC also found that Canada's position in this field has also been increased due to the research underway at TRIUMF. TRIUMF's accelerator division has contributed to development of significant knowledge about accelerators themselves and TRIUMF has also hosted major international conferences on accelerators, both of which contribute to elevating Canada's position in this field.

The IPRC also found that "Canada has one of the world-leading nuclear science programs, consisting of nuclear structure, nuclear astrophysics and fundamental symmetries" and highlighted TRIUMF's innovations and knowledge on the generation of intense beams of rare radioactive ions as leading-edge. The nuclear medicine research at TRIUMF is also considered world-class and some of the research underway has garnered international attention, including the work to develop an alternative production route for ^99mTc production via ¹⁰⁰Mo(p, 2n)^99mTc using accelerators. The nuclear medicine group is small; however, the IPRC noted that there are key researchers in the group who are recognized worldwide and help elevate Canada's position in this field. Further, the IPRC report states that "TRIUMF is Canada's brand in particle physics. This is the result of the standard of excellence maintained over the 45-year lifetime of the laboratory. In the past 5 years, the contributions of TRIUMF to the ATLAS experiment and to the LHC more broadly, have been internationally recognized." As the materials science group at TRIUMF is relatively small, much of their impact is as a result of the research conducted by their users. However, TRIUMF scientists have an international reputation for technical innovation in both instrumentation and sample environments.

The bibliometric study examined Canada's and TRIUMF's performance using several lenses, including examining Canada's ranking as one of the top 25 countries in terms of scientific output in this field. Canada had a moderate level of scientific output (ranked 10^th out of the top 25 countries); however, it achieved high rankings for both the scientific impact and research quality indicators. Canada ranked 2^nd overall for their scientific impact (ARC) and 3^rd overall for research quality (ARIF). This shows that while Canada does not produce as many papers as other countries which invest more heavily in this area, the Canadian contributions are of high quality and impact. ^{Footnote 17} The following table illustrates Canada's number of publications, scientific impact and research quality for subatomic sciences and each sub-field in which TRIUMF operates. The world average for each is 1.00; meaning that a score of less than 1.00 is less than the world average and above 1.00 is greater than the world average.

Table 6: Canada's Scientific Impact and Research Quality

Field	Scientific Output	Scientific Output Ranking Footnote 18	Scientific Impact (ARC)	ARC Ranking	Research Quality (ARIF)	ARIF Ranking
Particle Physics	9,771	11	1.54	4	1.19	7
Nuclear Physics	4,620	13	1.55	8	1.25	4
Accelerator Physics	2,126	11	2.18	6	1.25	5
Materials Science	1,867	10	1.24	4	1.12	6
Nuclear Medicine	4,355	8	1.43	3	1.24	2
Subatomic Sciences	16,752	10	1.55	2	1.23	3

Canada's positive reputation in this field is further corroborated by the report of the Canadian Council of Academies 2012 State of Science in Technology, where Physics and Astronomy was identified as one of six fields in which Canada excels, based on both bibliometric assessments and its rank in a survey of international experts. Further, the Council identified Nuclear and Particle Physics as one of nine sub-fields in which Canada leads the world in scientific impact, based on bibliometric data.

The positive reputation Canada has in this field and TRIUMF's contribution to it was also supported by the survey of TRIUMF's users, where 61% of respondents were international. In TRIUMF-related fields of research, 88% of respondents rated Canada's international reputation as being above average or world-class and 92% felt that TRIUMF had considerably or extremely contributed to Canada's positive reputation in these fields.

Key Finding 12: TRIUMF is central to much of the Canadian effort in subatomic physics.
The following image shows the Canadian network for subatomic sciences, with the sizes of the nodes proportional to the number of publications and the line thickness proportional to the number of co-publications. The colors represent the geographic locations of the institution (provincial or pan-Canadian).

Figure 5: Canadian Network in Subatomic Physics

Long description of Canadian Network in Subatomic Physics

 

This figure depicts a network map of Canadian organizations who publish is the area of subatomic physics. The figure illustrates the geographic location of each organization, the relative amount of publications for each organization, as well as the frequency of collaborations with other Canadian organizations. The central location of TRIUMF within the network map shows that TRIUMF frequently collaborates with other Canadian organizations and it emphasizes the central role the TRIUMF plays in the Canadian subatomic physics community. The network map also reveals that TRIUMF publishes at a rate that is comparable to, though slightly less than, Canada's most scientifically productive subatomic physics institutes (e.g. McGill University, University of Toronto, and University of British Columbia).

Source: Bibliometric Study of TRIUMF's Scientific Output, Science Metrix, 2013

Notes: Sizes of the nodes are proportional to the numbers of publications and widths of the linkages are proportional to the numbers of co-publications.

In Figure 5 above, the Canadian network for subatomic physics is shown to be very closely connected, with TRIUMF being one of the main central hubs. ^{Footnote 19} TRIUMF has strongest relationships with its member universities, (for complete list of member universities see Section 2.0: Profile) although the image shows TRIUMF also maintains relationships with other Canadian institutions and non-member universities. While the member universities shape TRIUMF's orientation, all researchers may apply to access the facilities. TRIUMF utilizes three external peer review committees who assess and evaluate proposals for access. The manner in which access is awarded was deemed fair by both interviewees and survey respondents.

TRIUMF's involvement in international activities strengthens its positive reputation outside Canada. All interviewees (Canadian and international) stated that TRIUMF has an excellent reputation as a contributor to key international projects. The survey also indicated that 92% of users either agreed or strongly agreed that TRIUMF is an international hub for science. This was reinforced through the peer review, which also identified TRIUMF as a hub for Canada's participation in a number of important experiments. Not only is TRIUMF the only entity within Canada with sufficient scale to provide adequate support in a number of specialised areas (e.g., accelerator construction and component engineering, detector construction, and large scale computing), TRIUMF's accelerators are the platform that enables much of the entrée for Canada to the major international programs.

Finally, the extent to which TRIUMF is involved in the overall Canadian effort for subatomic physics is illustrated through the proportion of NSERC funding for subatomic physics that is awarded to projects in which TRIUMF has a key participation. From 2008-09 to 2011-12, 74% of NSERC funding for this field went to projects in which TRIUMF is involved ($67M of $90.6M), showing that TRIUMF is central to much of the ongoing research in the Canadian community.

4.3 Social and Economic Benefits

This section examines TRIUMF's economic and social impacts in Canada. TRIUMF's expected socio-economic impacts have been reviewed taking into account the primary mandate of the facility to pursue scientific excellence.

4.3.1 Contribution to Economic Benefits

Key Finding 13:TRIUMF has contributed to the achievement of economic benefits by Canadian firms.

Examples of TRIUMF's activities in support of Canadian firms include: ^{Footnote 20}

• Producing medical isotopes to be sold by Nordion;
• Assisting Canadian companies (i.e., ACSI and PAVAC) in the development of specialized equipment or other competitive advantages (e.g., Tc-99m production technology), both to meet TRIUMF's needs and to expand their market share with sales to other international facilities and customers;
• Supporting Canadian companies in the development and commercialization of new technologies. Through AAPS, TRIUMF has aided in the establishment of one firm, IKOMED Technologies, which was spun off in 2012.
• Continued engagement with a previously spun off company (D-Pace) through technology licenses and subcontracted services; and
• Providing irradiation services to Canadian and multinational companies for testing products (i.e., MDA Corporation, Microsat Systems Canada Inc., Boeing and Cisco Systems, Inc.).

Over the past three years, about 75% of the contracts TRIUMF has awarded ($62M of $81M) have gone to Canadian firms, which keeps much of the federal (and other) funds invested in TRIUMF in the Canadian economy, as opposed to being spent abroad.

Key Finding 14: There are future economic benefits that will be realized through TRIUMF and AAPS activities, although improvements to TRIUMF and AAPS's commercialization practices could increase the number of future benefits.

It is expected that additional economic benefits may result from patents granted to TRIUMF during the evaluation period; however, the timelines for the realization of these benefits can be quite long. Between April 2008 and March 2013, 27 patent applications were filed and 28 patents were granted. ^{Footnote 21} Some notable examples include:

• Geological tomography using cosmic rays;
• System and method for the production of 18F-fluoride;
• Enabling technologies for the production of technetium-99m on small medical cyclotrons (patent pending); and
• Improved methods for Penning Trap mass spectroscopy (patent pending).

The peer review of TRIUMF included a review of the technology transfer and commercialization practices used by TRIUMF and AAPS, as well as the impacts of their commercialization activities to date.

AAPS was created in 2008 as one of eleven Centres of Excellence for Commercialization and Research and, as described in Section 2.0: Profile, acts as the commercialization arm of TRIUMF with the objective of expanding TRIUMF's societal and economic impacts in Canada. From an organizational perspective, the formation of AAPS was to create a singular focus of the commercialization core competence. Under Canada's Income Tax Act and CRA's policy regarding registered charities, TRIUMF (a registered charity) is limited in the use of its resources for the purpose of generating related business activity. A charity is restricted to use no more than 10% of its resources in the pursuit of related business activity and in TRIUMF's case, the commercialization of potential technologies. The creation of AAPS (a separate legal entity) allows TRIUMF to pursue its commercialization goals without the concern of reaching its allowed capacity – which, if achieved, would jeopardize its tax-exempt charity status. In addition, AAPS, being an incorporated non-profit organization, shields TRIUMF and its member universities from liability issues which may arise from commercialization activities. AAPS may also have the ability to access funding and partner with institutions in areas that would otherwise be restricted.

To ensure a clear demarcation between TRIUMF and AAPS, TRIUMF employees who provide services to AAPS (including TRIUMF senior management seconded to AAPS) bill their time to AAPS at appropriate market rates. Prior to 2013, AAPS operated relatively independently; more recently TRIUMF and AAPS have been working towards creating greater synergies and AAPS has been brought back under the umbrella of TRIUMF. In an effort to improve communication and coordination between the two entities, including maximizing cost efficiency, the head of TRIUMF's Business and Administration Division is also the president and CEO of AAPS (at 25% of seconded time). TRIUMF's chief financial officer (CFO) and head of Strategic Planning and Communication are other members of TRIUMF's leadership team seconded to AAPS.

Overall, the IPRC found that "TRIUMF has been able to support the Canadian industry to some extent; however, the lab has been more opportunistic than strategic in its commercialization-related activities. The recent realignment of AAPS with TRIUMF is expected to enable more strategic commercialization endeavors, although the IPRC proposes improved practices in this area." The IPRC noted that the recent restructuring to integrate AAPS more closely with TRIUMF provides a more effective commercialization platform to enable future benefits. The IPRC noted a list of best practices TRIUMF and AAPS should consider to improve the impact of their commercialization-related activities:

1. Prioritize potential areas of commercialization. This approach will allow TRIUMF to classify potential areas of commercialization into three tiers. Tier 1 would contain technology with short term potential, and Tier 2 and 3 would be classifications for more medium- to longer-term commercialization technologies. This approach will achieve several key objectives:
   1. Enable longer-term planning;
   2. Focus AAPS/TRIUMF resources; and
   3. Minimize distraction from important basic research.
2. Proactively target commercialization partners. The success of the technology transfer as well as the demand for AAPS resources will depend on the attributes of its commercialization partners. Some critical target attributes should be:
   1. Industry expertise; and
   2. Commercialization skills and resources.
3. Clearly define objectives of the "deal". The process of defining objectives of the specific commercial effort will assist AAPS to structure and negotiate a win-win arrangement within the context of maximizing monetary return or achieving more strategic imperatives.
4. Quantify targets, develop specific business plans and measure results. This process will focus AAPS on achieving its targeted objectives and adopt a process of continuous improvement.
5. Ensure continued industry/stakeholder consultation. This approach will provide valuable insights into industry requirements as well as enhance stakeholder commitment to TRIUMF technology.
6. Develop a solid patent strategy. Often the value of a technology transfer is assessed on secured intellectual property (IP). AAPS/TRIUMF should predetermine its strategy for each of its commercialization efforts to ensure that publications do not jeopardize the retention of IP value for Canada.

Key Finding 15: TRIUMF's short-term ROI to Canada is estimated to be at least 1.7 times the public investment, which is significant considering its fundamental research mandate.

"While the primary mandate of TRIUMF is the pursuit of excellence in fundamental research into subatomic physics, and not the pursuit of economic impacts, such impacts have been significant."

- Return on Investment in Large Scale Research Infrastructure, Hickling Arthurs Low (HAL), 2013.

In a recent economic impact study, conducted by Hickling Arthurs Low Corporation (HAL), the short-term rate of return on public investment in TRIUMF was estimated to be 1.7 dollars for every dollar invested. It is the public expenditures that have generated the commercial revenues and other economic activity that represent the 1.7 rate of return. This calculation considers the past ten years of TRIUMF's activities and expenditures, as well as certain expenditures from AAPS, relevant business enterprise revenues from four related companies (Nordion, D-Pace, PAVAC, and ASCI), and economic activity that was generated by visitors who attended conferences hosted at TRIUMF. The rate of return represents the benefit of TRIUMF in economic terms only and does not include a number of important considerations, including: social, health, and environmental impacts; benefits to users of transferred technology (spin-offs) and other technical considerations. On balance, these exclusions render the 1.7 rate of return as being a conservative estimate of impact.

4.3.2 Contribution to Social Benefits

Key Finding 16: TRIUMF's research has led to several health benefits, the most notable being the production of medical isotopes using non-reactor technologies.

A number of evaluation methods identified social benefits that have been realized as a result of TRIUMF's past activities, specifically within the nuclear medicine group. One example is the treatment of 10-12 patients per year for a cancerous growth on the back of the eye, called choroidal melanomas. Before proton treatment became available, the most common course of action was removal of the eye. Now patients can retain a useful vision because of TRIUMF's intervention. TRIUMF is also working to adapt their muon technology for detection uses at Canadian borders and ports, which will contribute to greater security and safety for passengers. The IPRC also highlighted TRIUMF's research in nuclear imaging techniques for the treatment of neurological disorders, including Parkinson's, as a significant opportunity for TRIUMF to contribute to the understanding and treatment of these disorders and a benefit that could be realized in the future.

The most significant, and well-known, social benefit to come from TRIUMF in the past five years is the production of medical isotopes using non-reactor technologies and the associated medical and environmental benefits. Medical isotopes are an important component of the Canadian and worldwide health care systems. Technetium-99m (Tc-99m) is the most commonly used medical isotope, accounting for approximately 80% of nuclear medicine diagnostic procedures. Canada had been a main producer of this isotope, using an aging nuclear reactor based in Chalk River, although it faced several disruptions in its production in 2007 and 2009. As Tc-99m has a very short half-life (only 6 hours), ongoing supply of the isotope is critical to support medical procedures. Following these disruptions, the Canadian government announced it would investigate alternative methods of production and the reactor would not produce medical isotopes after 2016. ^{Footnote 22}

NRCan's Isotope Technology Acceleration Program (ITAP) provided funding to a small number of Canadian research organizations including to a consortium led by TRIUMF, to pursue the production of Tc-99m using cyclotrons instead of reactors. In February 2012, the research team proved that Tc-99m could be produced using existing medical cyclotrons in BC and Ontario. In June 2013, the research team successfully produced enough Tc-99m in one six-hour shift to support a metropolitan area like Vancouver, demonstrating that cyclotron-based production of the isotope may be sufficient to meet the demand. ^{Footnote 23}

Key Finding 17: TRIUMF has contributed significantly to the development of the Canadian subatomic physics community through both its training programs and its role in attracting researchers to Canada.

The IPRC and the interviewees were very impressed with the development and training of students at TRIUMF. The IPRC reported that "TRIUMF provides exceptional training to the next generation of scientific leaders, as a result of both the detailed experiences of individuals at TRIUMF and the overall scope of the program." The IPRC highlighted the very large number of students hosted, particularly when compared to the limited number of research scientists at TRIUMF and that this "represents an excellent leveraging of limited numbers of more senior staff." The diversity of student training programs at TRIUMF was commended as was the quality of the experience received by the students, including much more hands-on training and extensive responsibilities than would be experienced elsewhere. The IPRC found that its training of students and postdocs had contributed to "an amazing hiring record after leaving TRIUMF, landing in good positions at institutes and professorships at notable universities". The accelerator group was singled out by the IPRC for the unique training opportunities available at TRIUMF and noted that "[t]here are only a very few places in the world that give this quality of education in accelerator science and technology." Both the nuclear medicine and materials science groups were also highlighted as having a particularly high number of students given the small size of each group.

Over the evaluation period (2008-09 to 2012-13), TRIUMF's researchers supervised a total of 463 students completing their graduate degrees or oversaw the training of post-doctoral fellows (PDFs). This number includes 113 M. Sc., 173 PhD and 177 PDFs. In the year 2012-13 alone, 35 international PDFs came to work at TRIUMF. Countries of origin included Germany (6), United States (5), India (4) and the United Kingdom (3). All interviewees believed that TRUMF was of great benefit for Canadian and international students. According to some interviewees, without TRIUMF, students would not have had the experience of exposure to world-class equipment and techniques. Some interviewees also felt that, in the absence of TRIUMF, there would be fewer students venturing into this discipline, and consequently fewer Canadian students trained in subatomic physics.

"The science, leadership and people at TRIUMF have enabled TRIUMF and other Canadian universities to attract some of the best scientists to their labs."

- External interviews, 2013.

TRIUMF's reputation is illustrated through its ability to attract and retain high quality students and researchers to itself and its member universities. The results of the survey of TRIUMF users revealed that TRIUMF had contributed 'considerably' or 'extremely' to their decision to either move to Canada (67%) and/or to remain in Canada (72%) for the conduct of their research. Users of TRIUMF's facilities also believe that TRIUMF has enabled them to develop a new skill set or expertise (87%). As well, 93% of users agreed or strongly agreed that TRIUMF has enabled them to make significant contributions to their field of research.

Key Finding 18: TRIUMF's outreach activities are appropriate and have contributed to the elevation of public awareness of TRIUMF and subatomic physics in Canada.

Most interviewees felt that TRIUMF's outreach activities had been successful, with many attributing TRIUMF's progress in this area over the past seven years to the focus and direction provided by the management team. While a few external and internal interviewees noted that outreach can always be improved and increased, they acknowledged that this requires resources and felt that additional outreach should not be undertaken at the expense of research activities. TRIUMF has worked collaboratively with other stakeholders, including those in the surrounding geographic region (e.g., Genome BC), as well as those involved in the same scientific community (e.g., SNOLAB, Perimeter Institute). This collaborative behaviour was noted as a best practice because it has contributed to generating critical mass and increased the impact of individual efforts.

When compared to other facilities (international and domestic) TRIUMF's performance in outreach appears to be similar in scope, yet limited in the amount of resources ($30,000-$50,000 yearly) devoted to it. Some of TRIUMF's outreach activities include producing an animated physics and education video available to Canadian high schools as a teaching aid, hosting field trips for about 660 high school students annually, engaging in online and social media outreach, as well as providing tours for the general public.

4.4 Unintended Outcomes

TRIUMF's activities do not appear to have resulted in unintended negative outcomes. In regards to positive unintended outcomes, some stakeholders argued that TRIUMF's development of novel techniques for producing medical isotopes and their partnership to establish a heat-recovery/district-energy system with UBC were both positive and unintended – due to the fact that they involved the identification of opportunities outside of TRIUMF's typical scope. Other stakeholders disagreed with this characterization, however, and argued that these outcomes are linked to TRIUMF's existing or proposed activities.

5. Findings: Resource utilization

5.1 Economy and Efficiency

This section examines the extent to which TRIUMF has been able to generate outputs and progress towards its outcomes in an economic and efficient manner.

Key Finding 19: TRIUMF does not have a comprehensive performance management system in place.

During this evaluation TRIUMF has developed a logic model. It has not however identified performance measures or targets dates for its outputs or outcomes, nor does it track the costs of the activities to produce its outputs. (See Appendix C: Logic Model for the list of TRIUMF's outputs). In addition, it is not clear how and when the data will be collected and if a formal process will be established to review the performance management system.

For instance, TRIUMF does not have an effective system for tracking the number of invited talks and conferences given by TRIUMF scientists. Without a doubt, the tracking of outputs of research is inherently difficult to do as is assessing the cost of the operations that support scientific discoveries. Still, without this information it is more difficult to produce a comprehensive evaluation of TRIUMF's economy and efficiency and show areas for improvements to better optimize resources. This information is also important to also better demonstrate the impact of the public investment in TRIUMF and value-for-money. As TRIUMF is setting up a new management system there is an opportunity to capture this information.

Key Finding 20: TRIUMF has been able to generate outputs and progress towards their outcomes despite the dual challenge of increased input price pressures and a static level of operational funding from NRC. Concerns were raised around the lack of critical mass in two of TRIUMF's areas of research, the on-going maintenance of the infrastructure of the facility and its' ability to continue to sufficiently support operations once ARIEL comes online.

When assessing a program's efficiency and economy it is important to first consider the contextual factors that may have affected cost-effectiveness. In the case of TRIUMF, a number of external factors influenced the costs they paid for inputs as well as their level of operational funding. Internal sources identified a number of these, including:

• Inflation (average of 1.8% per year over the evaluation period);
• Increased energy costs;
• Increased speciality equipment costs and aging infrastructure;
• Changes in provincial tax legislation costing more than $1M per year; and
• Increased value of the Canadian dollar, compared to the US dollar.

These external factors, with the exception of the increased Canadian dollar value, raised the cost of inputs for TRIUMF and created challenges for operating within their static level of operational funding.

Despite these external challenges however, TRIUMF has managed to control costs well. As is shown in Figure 6, TRIUMF's expenditures on operations, salaries, and benefits have had a modest (1.3%) average annual growth rate (in real, inflation-adjusted terms) over the last ten years and an almost flat rate of growth (-0.05%) over the evaluation period.

One particular external factor – inflation – has also had an impact on the amount of funding that TRIUMF receives in real terms for operations. Over the evaluation period, inflation (as measured by the Canadian consumer price index (CPI)) was 6.7%. Over the ten year period of the two contribution agreements, this amount was 18.4%. This means that, when adjusted for inflation, NRC's static level of funding - of around $40 M annually for the last ten years – is declining in real terms, as depicted in Figure 6. Given that this funding is mostly for operations, this inflationary impact has resulted in a decreased amount available for day-to-day operations. However, TRIUMF's ability to sustain progress towards their outcomes may be compromised when their decreasing ability to pay is considered along with the increasing input cost pressures. Still, to date, TRIUMF has been able to manage these inflationary pressures well.

Figure 6: Comparison of TRIUMF's Operational Budget to NRC's Funding (in Real, Inflation-Adjusted Terms)

Long description of Comparison of TRIUMF's Operational Budget to NRC's Funding (in Real, Inflation-Adjusted Terms)

Year	2003-04	2004-05	2005-06	2006-07	2007-08	2008-09	2009-10	2010-11	2011-12	2012-13
Operational expenditures + Salaries and benefits	$49,777,309	$49,433,911	$54,183,769	$52,984,145	$59,063,146	$54,507,906	$56,077,372	$56,084,997	$58,476,224	$58,668,531
NRC funding	$38,910,506	$38,204,394	$41,121,495	$41,704,858	$46,188,341	$38,124,452	$38,461,538	$38,626,609	$36,697,248	$36,154,478
Salaries and benefits	$30,895,455	$31,532,786	$32,108,174	$31,290,826	$32,102,881	$32,111,041	$35,054,679	$34,575,995	$33,851,772	$33,224,566
Operational expenditures	$18,881,854	$17,901,125	$22,075,595	$21,693,319	$26,960,265	$22,396,865	$21,022,692	$21,509,002	$24,624,452	$25,443,965

Sources: TRIUMF Data; Statistics Canada

Over the evaluation period, the following measures helped TRIUMF to achieve this low rate of growth in expenditures and compensate for the declining amount of NRC funding (in real terms) for operations:

• Restructuring and cutting staff;
• Maintaining a low rate of economic salary increase;
• Reducing funds towards employee recognition;
• Deferring infrastructure maintenance;
• Reducing the number of board meetings from 4 to 2 annually and creating an executive board committee; and
• Reducing travel rates, under TRIUMF's travel and hospitality policy, to a rate below comparable Government of Canada travel rates.

These measures have resulted in at least $2.4 M in savings per year. Despite these reductions, TRIUMF managed to maintain a strong level of outputs and they also made strong progress towards their outcomes, as was shown in the performance section. This is a strong indicator of TRIUMF's efficiency and ability to economize.

This efficiency is further confirmed by the user survey and external interviewees. Almost all users found that TRIUMF was efficient, as is shown in the table below.

Table 7: Selected Survey Responses to Questions on TRIUMF's Efficiency

Survey Question	Disagree or Strongly Disagree (% of Respondents)	Agree or Strongly Agree (% of Respondents)
The process for applying for access is reasonably straightforward	4%	91%
Decision-making for access was done in a timely manner.	7%	89%
The staff at TRIUMF ensured that my research could be conducted efficiently	3%	97%

Source: User Survey

Note: Row percentages may not add up to 100% because the 'not applicable' data from each question is not represented.

In addition to the survey, all external interviewees considered that TRIUMF had progressed significantly towards their long-term goal (of being a leader in subatomic physics research) within the context of its budget (~$72 M annually). Some notable comments included, TRIUMF is "lean and mean," "[it] punches above its weight," "[it] is doing it on a shoestring while other require the whole shoe," and it represents a "good value."

While most evaluation evidence points to increased efficiency during the evaluation period, there were a number of sources that raised concerns about the present and longer term impact of TRIUMF's efforts at economization in operations. For example, one key external interviewee noted that several of TRIUMF's measures to reduce input costs may actually be false economies, as providing uncompetitive pay raises and reduced bonuses may lead to a loss of key personnel. Still others were concerned that existing areas of research, especially in material sciences and nuclear medicine do not have the full complement of staff needed to conduct the research.

"The labs are close to the edge in terms of operations. TRIUMF has a full spectrum of activities but may not have the critical mass needed. They have continued to pursue ambitious projects with flat funding."

- External Interviewee, 2013.

Multiple evaluation sources noted that deferred maintenance on aging equipment could also lead to lower subscription rates and a reduced number of publications. An example provided was the shutdown of one of TRIUMF beamlines, which has decreased the amount of beamtime available for specific research areas, most notably in the area of materials science.

Perhaps the most significant concern is TRIUMF's ability to sustain a satisfactory level of operations and maintain its unique facility capabilities and competencies once new infrastructure comes online. For example, the NSERC Long-Range Plan 2011-16 stated that the "completion and operation of ARIEL beyond 2016 will limit TRIUMF's ability to support major subatomic physics initiatives without an increase in federal funding." As well, one surveyed user commented: "The continued lack of operating funds (rather than capital) is a cause for concern - there has been significant investment in infrastructure and an expansion of lab activities over the last decade or more without a proportional increase in staff or operating budgets."

This area of concern is strongly linked to TRIUMF's otherwise strong success in attracting funding for capital projects and other sponsored activities from sources other than NRC. As shown in Figure 2, TRIUMF has been able to attract a significant amount of additional funding from competitive funding sources as well as a relatively consistent amount of revenues from private companies, non-profit organizations, and international partners (through collaboration agreements and commercial revenue).

Figure 7: TRIUMF Funding and Revenue, by Source

Long description of TRIUMF Funding and Revenue, by Source

TRIUMF Financial Information	2008-09	2009-10	2010-11	2011-12	2012-13
NRC Contribution Agreements	$43,500,000	$44,000,000	$45,000,000	$44,000,000	$44,000,000
CFI/BCKDF	$1,114,884	$1,946,096	$5,580,147	$10,196,335	$20,361,685
NSERC	$5,970,896	$6,587,586	$6,309,557	$6,316,503	$6,578,602
Other*	$0	$918,968	$1,332,262	$5,921,711	$6,273,155
Nordion	$4,370,636	$4,350,138	$4,219,420	$4,190,636	$3,892,558
Non-Profit Organizations / International Partners	$2,092,220	$2,191,470	$2,110,304	$2,207,629	$2,664,758
AAPS	$1,033,968	$1,166,618	$1,754,608	$1,806,141	$1,512,691
Intramural Accounts	$1,801,083	$1,442,797	$1,275,087	$1,116,365	$1,039,663
Commercial Revenue & Interest	$1,704,792	$2,399,154	$2,726,731	$1,642,185	$1,361,202

Source: TRIUMF Data

These supplementary funds, as a whole, increased 142% over the evaluation period and are solely responsible for TRIUMF's total revenue growth of 42% over the same period. From the perspective of NRC, TRIUMF's additional funding and revenue can help to leverage the impact of NRC's investment, as TRIUMF is able to support and participate in important experiments and expand its facilities and capabilities. While this additional funding has likely allowed TRIUMF to progress more quickly towards its goals and enhance the level of attainment of these goals during the evaluation period, there are also signs that the activities associated with funding have begun to strain TRIUMF's operational resources, as TRIUMF has been required to commit resources towards the day-to-day operations that support them.

Specifically, concerns were raised regarding TRIUMF's ability to continue to sufficiently support operations once ARIEL comes online (which is expected to occur in 2014). While ARIEL is expected to deliver significant benefits to TRIUMF (e.g. scientific outputs and productivity) there remains the possibility that the operational budgetary strain may impact TRIUMF's ability to support existing infrastructure, research areas, and other sponsored activities. In turn, this strain may begin to limit TRIUMF's ability to deliver outcomes related to all existing research areas. In other words, TRIUMF may be unable to deliver a level of impact that is commensurate with the increased amount invested (which is exclusively from sources other than NRC).

Alternative approaches

When assessing the economy and efficiency of a program, it is helpful to consider possible alternative approaches described below in program design, delivery, and administration. These approaches may represent a greater level of economy or efficiency in themselves or they may be means by which greater economies or efficiencies may be identified. Below possible alternatives are discussed.

Key Finding 21: Costs related to NRC's administration of its contribution agreement to TRIUMF are very low.

In general, one possible way of achieving greater economy is to more efficiently and economically administer resources to the program. TRIUMF's main mechanism for funding is their contribution agreement with NRC (proposed and revisited every five years). This evaluation has found that this mechanism, as opposed to funding received from competitive sources, is very efficient and economical. Fewer resources are needed by NRC than is normally required to administer funding through competitive review processes. Over the evaluation period, the cost to administer NRC's contribution agreements with TRIUMF was $726K or an average of $145K annually. This cost results in an average operating cost ratio of 0.33% (total NRC expenses divided by total NRC funding to TRIUMF), which was found to be lower than the cost of administering competitive funds.^{Footnote 24} The expenses for the current year (2013-14) for the renewal of the terms and conditions of TRIUMF are not included. As the costs based on time will be systematically reported through Sigma ^{Footnote 25} , it is expected that more precise information will be available on the overall cost to administer TRIUMF.

This is the first evaluation of TRIUMF and the evaluation cost will exceed the cost of previous peer reviews. The current evaluation has undertaken to examine multiple lines of evidence in addition to scientific excellence such as operating efficiency and economic impacts, creating a baseline of data against which TRIUMF can be measured in future evaluations.

Interviewees had different views on the appropriateness of the level of NRC costs in administering TRIUMF's contribution agreement. A few external interviewees expressed concerns that this cost is very low and felt that NRC should devote more resources in the oversight of TRIUMF. However, some internal interviewees to TRIUMF disagreed and felt that TRIUMF's governance (discussed further in Section 6.0: Governance) has sufficient checks and balances in place.

Key Finding 22: TRIUMF is more efficient than other international facilities and programs operating in similar fields.

The comparison study of large international subatomic physics facilities found that TRIUMF operates with a much lower amount of funding than its peers, yet is able to achieve a comparably high level of performance. For example, as shown in Figure 8 below, the 2011-12 base operating expenditures for TRIUMF were nearly four times smaller than the average of five comparable facilities yet, as the performance section revealed, TRIUMF was able to compete strongly with these labs - in terms of scientific quality - and generate a significant amount of impact

"[TRIUMF] is doing it on a shoestring while others require the whole shoe"

- External Interviewee, 2013.

As well, TRIUMF's presence, reputation, and direct collaborations with the international physics communities have allowed Canadian subatomic physicists the opportunity to participate in the most important international experiments. As a result, from an international comparison perspective, TRIUMF appears to be a relative bargain and the alternative approaches pursued in other countries will likely not lead to more economical outcomes.

Figure 8: Base Operating Expenditures of Comparable Facilities (2011-12)

Long description of Base Operating Expenditures of Comparable Facilities (2011-12)

Facility	Base operating expenditures	Average operating expenditures
Paul Scherrer Institute (Switzerland)	$343,084,365	170524755.1
Fermilab (USA)	$329,780,589	170524755.1
Hevesy-Riso (Denmark)	$123,212,375	170524755.1
Jefferson Lab (USA)	$109,410,495	170524755.1
Rutherford Appleton Lab* (UK)	$73,660,707	170524755.1
TRIUMF (Canada)	$44,000,000	170524755.1

Source: 2013 Large Scale Facility Comparison Study – Self-Reported Survey of Facilities

* As opposed to the other organizations in this figure, the data for Rutherford Appleton Lab (RAL) does not represent the operations of the whole laboratory but rather an estimate of operations expenditures in areas similar to TRIUMF (which was estimated in collaboration with a director at RAL). This was required due to the unique organizational structure of RAL and its associated reporting requirements.

6. Findings: Governance

As previously indicated, all financial and administrative responsibilities for TRIUMF are assumed by the full member universities. The TRIUMF Board of Management is responsible for the supervision and control of TRIUMF, and is composed of representatives from the eighteen full and associate member universities, as well as two private sector members appointed by the board. TRIUMF's Director has the overall responsibility of day-to-day operations as well as the development and implementation of its science programs. The Director reports to the Board of Management through the board Chair. The Board of Management has various standing committees that deal with specific issues relevant to TRIUMF's operations including: Audit Committee, Executive Committee, Finance Committee, Personnel and Administrative Committee, Safety and Security Committee, and Innovations and Industrial Partnerships Committee. Additionally, TRIUMF's Private-Sector Advisory Committee (PSAC), which is made up of prominent business leaders and some federal and provincial government participants, and provides advice to the director.

In order for TRIUMF to deliver on its mandate, two corporations were created: TRIUMF Accelerators Inc. (TAI) and Advanced Applied Physics Solutions, Inc. (AAPS) (described in Section 4.3.1). Formed in 2006, TAI is a not-for-profit corporation that holds the operating license for TRIUMF. TRIUMF requires an operating license from the Canadian Nuclear Safety Commission (CNSC) to operate its accelerators. As TRIUMF is a joint venture owned by a group of universities, there is no sole owner to be issued the operating license and to be held accountable for the operation of the accelerators. The creation of TAI allows TRIUMF to comply with the CNSC regulations and legally operate. The TAI Board of Directors includes the representatives from each of the eleven full member universities. The Director of TRIUMF is also the President and CEO of TAI, and the senior positions in the company include some members of the TRIUMF leadership team.

NRC has put in place several mechanisms that support the oversight of TRIUMF. The Advisory Committee on TRIUMF (ACOT), a panel of prominent international experts, advises NRC on aspects related to the determination and administration of the federal contribution to TRIUMF, and provides scientific program advice to the Director of TRIUMF. The Agency Committee on TRIUMF (ACT), whose membership includes representatives from NRC, NSERC, and Industry Canada, presides over two main responsibilities: overseeing Government of Canada investments in TRIUMF and monitoring the economic benefits derived from this investment. ACT reports directly to the President of NRC. A final mechanism of oversight is NRC's appointment of an international peer review committee. This appointment occurs every five years to review and evaluate TRIUMF's scientific performance as well as their five-year strategic plan proposals. The IPRC referenced in this evaluation is an example of this oversight mechanism.

6.1 Governance Best Practices

Key Finding 23: TRIUMF's governance model includes many best practices for governance.

While TRIUMF's governance model was not designed to meet CFI's Oversight Framework for Major Science Initiatives,^{Footnote 26} it follows many of the best practices described in the framework, including:

• A flexible, adaptable model that can evolve as the facility and its needs change over time;
• Delegation of specific responsibilities to sub-committees;
• An appropriate number of board meetings annually, including at least one face-to-face meeting;
• Regular in-camera sessions, where TRIUMF management is not present, in order to maintain board independence; and
• Board members who are knowledgeable, effective, and independent, avoiding actual or perceived conflicts of interest.

The IPRC and almost all internal and external interviewees considered TRIUMF to have an appropriate internal governance structure. However, as discussed below, suggestions for improvements to the ACT were made.

6.2 Opportunities for Improvement to TRIUMF's Governance Model

Key Finding 24: The ACT has not been used effectively to coordinate TRIUMF's activities.

Most suggestions proposed by interviewees regarding the governance model of TRIUMF were relatively minor. A few external interviewees noted that the size of the board (i.e., 24 voting members) can be a hindrance to decision-making and approvals. However, the board has implemented an executive committee made up of six members to address this issue. In addition, the IPRC considered that the increasing number of board members and associate members "demonstrates real broad-based national engagement in TRIUMF".

While the role of ACT is to act as a coordination mechanism among the federal granting agencies, NRC and Industry Canada, interviewees had differing perceptions of the role ACT should play. The Terms of Reference for ACT are dated (November 2000) and are very general in relation to the role and responsibilities of ACT.

The potential roles interviewees felt ACT should play ranged from decisions in support of TRIUMF's funding renewal and coordination of agencies' investments to discussions on cross-institutional issues relating to 'Big Science' including those broader than TRIUMF. For example, ACT could play a role in ensuring that the various funding sources are aligned so that more capital is not awarded to projects than the operating costs that the NRC and NSERC budgets can support.

Some interviewees noted that ACT was not being used effectively as it has not met in several years and documentation states that ACT typically meets twice annually. One key interviewee felt the committee was not needed at all and another felt it could be merged with the NRC advisory committee for astronomy as almost all of the committee members participate in both committees.

Evaluation participants from the physics community acknowledged that efforts have been made to coordinate federal agencies involved in funding TRIUMF through the establishment of ACT. Still, increased interagency discussions on funding for physics projects were deemed necessary by some evaluation stakeholders. The membership of the ACT should also be reviewed to include key funding organisations of TRIUMF. For example, the CFI is not a member of the ACT, yet it contributes significantly to the overall capital investment at TRIUMF. The IPRC recommended that "NRC should convene the Agency Committee on TRIUMF (ACT) early in 2014 to communicate and better coordinate the plans for TRIUMF's next period of activity".

6.2.1 Governance of Other International Physics Facilities

Key Finding 25: TRIUMF's governance structure has advantages over those used by other international physics facilities.

Table 8: Comparison of Governance Models

Models	Facilities	Characteristics
Owned and operated by a non-incorporated consortium of universities	TRIUMF	5-year stable funding; No political influence in five year strategic planning exercise, yet a contribution agreement needs to be negotiated and approved by NRC; Limited bureaucratic processes; and Trimester reporting to the governmental agencies and the Board.
Government owned, contractor operated by universities (GOCO)	Fermilab (USA) Jefferson Laboratory (USA)	Annual federal budget approval; Government sets overall objectives and provides close oversight; Very bureaucratic; and Large accountability requirements.
Government owned and government operated (GOGO)	Rutherford Appleton Laboratory (UK	Funding is allocated in four year periods; Long-term lab objectives and priorities are signed off by a governing body (Council) that contains members of their 'parent' government department. Yearly and term objectives are set by and scrutinized by parent department; Some bureaucracy; and Semi-frequent mechanisms for accountability (e.g., reporting every six months to parent government department).

Source: Large Scale Facility Comparison Study

In contrast to models used to manage large scale physics facilities in other parts of the world, the TRIUMF model appears to lend itself to a more independent pursuit of scientific avenues with less political influence. Freedom of inquiry is perceived by scientists as an essential element to preserve. Moreover, the stable funding over a five year period offered through NRC's contribution agreements allows for continuity of activities and longer-term planning. The TRIUMF model may however provide less public accountability to federal government as it is outside of direct central government control. The model does provide enhanced accountability to the TRIUMF Board of Management and the funding agencies.

"The bureaucracy at TRIUMF is very minimal and they are nimble compared to other international facilities."

- External interviews, 2013.

7. General Conclusions

This evaluation demonstrates that TRIUMF meets the needs of the subatomic physics community in Canada and internationally. Users depend on TRIUMF's unique facilities to carry out their research and in its absence most of Canada's subatomic physics research would need to be conducted elsewhere in the world. The relevance to the community is evidenced through the high subscription rates to TRIUMF facilities as well as by the role it plays internationally in key physics projects. NRC was found to have administered well the transfer and the monitoring of funds to TRIUMF; there was no alternative funding agent that was deemed more appropriate.

The findings have also confirmed that TRIUMF delivered excellent scientific output in all of their areas of research. Internationally, TRIUMF is seen as a scientific leader and collaborator. Without continued financial support to ensure the sustainability of TRIUMF's research program, TRIUMF would certainly not be able to retain this leadership role in physics for Canada nor would Canadian scientists readily have access to international physics facilities.

TRIUMF also contributes economic benefits for Canada. Its contribution is deemed significant for an organization whose mandate is primarily to conduct basic science. Nevertheless there are best practices that could be applied to its commercialization related activities that would yield greater results.

In addition, TRIUMF was found to operate with economy and efficiency. Still, concerns were raised regarding TRIUMF's ability to continue to operate within existing levels of funding. Evidence suggesting this concern includes equipment and facilities that have fallen into disrepair and an insufficient number of human resources available to conduct the planned research program in the areas of nuclear medicine and materials science. These signs of operational budget strain have prompted concerns about TRIUMF's ability to adequately support its main competencies and current and future infrastructure investments without sacrificing support to those programs that are currently supported through soft funding. In turn, this strain may limit TRIUMF's ability to maintain scientific excellence in areas that have been supported through targeted programs (from sources other than NRC).

Since TRIUMF relies on competitive funding from a number of organizations for a significant portion of their revenue, the lack of coordination among the funding organizations of TRIUMF adds to the level of uncertainty at TRIUMF and its ability to plan for the long term. In this regard, the Agency Committee on TRIUMF should enhance its leadership role in the coordination of TRIUMF's activities.

Overall, the findings of the evaluation of NRC's contribution to TRIUMF show that the activities undertaken by TRIUMF represent good value-for-money for NRC and for Canada. The implementation of the recommendations will be important to enable TRIUMF to continue to perform at a very high level, yield greater impacts from their commercialization activities and ultimately support the priorities of the research community in subatomic physics research and Canada's S&T strategy.

8. Management Response

Recommendation	Response and Planned Action(s)	Proposed Person(s) Responsibilities	Timelines	Measure(s) of Achievement
Recommendation 1: NRC should continue to provide stewardship to TRIUMF for the benefit of Canada's research community and in support of the government of Canada's S&T strategy.	Recommendation accepted. NRC will continue to provide stewardship to TRIUMF by managing TRIUMF's Contribution Agreement and by maintaining the Advisory Committee on TRIUMF (ACOT) which provides advice on scientific and technological issues related to the facility. NRC will continue to participate in an ex officio capacity on TRIUMF's Board of Management and on the Board's Audit Committee to ensure the continued coherence of the governing bodies administering TRIUMF and the responsible management of resources.	VP Emerging Technologies – NRC	Ongoing	NRC continues to provide stewardship to TRIUMF by managing their contribution agreement, maintaining ACOT, and participating (in an ex officio capacity) on TRIUMF's Board of Management and Audit Committee.
Recommendation 2: NRC should encourage TRIUMF to be more strategic with its commercialization activities including investigating and adopting best practices.	Recommendation accepted. NRC and ACOT have expertise and experience that can assist TRIUMF in the area of industrial partnerships and commercialization. NRC will work with TRIUMF on a Business Development Plan which will outline measurable goals and targets for the commercialization of TRIUMF technologies and the adoption of other best practices to improve the overall impact of commercialization activities. A draft of this Plan will be presented to ACOT and the Plan will be finalized as part of entering into Five-Year Plan 2015-2020.	VP Emerging Technologies - NRC Director – TRIUMF ACOT	April 2014-15	TRIUMF, NRC, and ACOT investigate best practices in commercialization. Business Development Plan is developed with input from NRC and ACOT (by April 2015).
Recommendation 3: To support NRC in promoting the value and accountability of TRIUMF, TRIUMF should define and implement an effective performance management system that is aligned with its strategic objectives and operational objectives.	Recommendation accepted. TRIUMF already has a logic model that lays out the relationships between the resources, activities, outputs and outcomes of its activities. NRC will work with TRIUMF to define key performance indicators that can be tracked over time in a consistent manner in order to measure TRIUMF's progress in key activities. The framework for performance indicators will be discussed with ACOT in the 2014-2015 fiscal year and will be incorporated into TRIUMF's annual reporting process by 2015-2016.	VP Emerging Technologies – NRC Director – TRIUMF	2016	TRIUMF defines trackable key performance indicators (KPIs). KPIs are included in TRIUMF's annual report for 2015-16.
Recommendation 4: NRC should ensure that TRIUMF has an appropriate mechanism in place enabling them to prioritize and manage the ongoing maintenance of infrastructure at the facility.	Recommendation accepted. NRC will work with TRIUMF to ensure that a Risk Assessment Framework is in place that will assist in identifying maintenance issues and prioritizing reinvestment in the facility. The Risk Assessment Framework will be discussed annually with ACOT.	VP Emerging Technologies - NRC Director – TRIUMF ACOT	Starting 2016	Risk Assessment Framework is in place. Risk Assessment Framework is discussed annually with ACOT.
Recommendation 5: NRC should ensure that TRIUMF has a plan in place to prioritize its financial and human resources to achieve critical mass in its areas of research.	Recommendation accepted. Working with TRIUMF Management, NRC will ensure effective mechanisms are in place that will enable TRIUMF to prioritize across key initiatives, allowing it to function most effectively within its allocated budget. As the laboratory begins its 2015-2020 performance period, an operating plan reflecting these priorities will be presented to ACOT and ACT.	VP Emerging Technologies - NRC Director – TRIUMF	2015	Mechanisms are in place for TRIUMF to prioritize across key initiatives. TRIUMF's first operating plan for the 2015-2020 period reflects these priorities.
Recommendation 6: NRC should convene the Agency Committee on TRIUMF to discuss and better coordinate the plans for TRIUMF. Furthermore, the expected role of the ACT should be clarified as well as its membership reviewed through an update of its Terms of Reference and communicated to stakeholders to ensure a consistent understanding of its role.	Recommendation accepted. A meeting of the Agency Committee on TRIUMF (ACT) will be convened early in FY 2014-15 to discuss TRIUMF's Strategic Plan and the Evaluation of the facility as well as future directions for ACT.	VP Emerging Technologies - NRC	2014-15	ACT convenes a meeting in early FY 2014-15.

Appendix A: Evaluation Matrix

Questions	Methods
	Document Review	Data Analysis	Peer Review	Comparison of Select Physics Facilities	Key Informant Interviews	User Community Web Survey
Relevance
R1. Continued Need for the Program
1. Does TRIUMF continue to address a demonstrable need in supporting Canadian subatomic physics? What is the role of TRIUMF in supporting this scientific community?	✓	✓	✓	✓	✓	✓
R2. Alignment with Government Priorities
2. Is the program aligned to federal government priorities in the area of science, technology, and innovation, and to NRC's strategic outcomes?	✓	✓
R3. Alignment with Federal Roles and Responsibilities
3. Is NRC's current role as contributor to TRIUMF appropriate, given the roles and responsibilities of the federal government overall and other federal government agencies involved in funding components of TRIUMF?	✓	✓	✓	✓
Performance (Effectiveness, Efficiency and Economy)
P1. Achievement of Expected Outcomes
4. To what extent has TRIUMF-supported research generated scientific benefits to the Canadian and worldwide communities?	✓	✓	✓	✓	✓
5. To what extent has TRIUMF contributed to social and economic benefits for Canada?	✓	✓	✓
6. To what extent has TRIUMF enhanced and/or sustained Canada's position among the world's leaders in physics?	✓	✓	✓	✓	✓
7. Have there been any unintended (positive or negative) outcomes as a result of TRIUMF?	✓	✓	✓
P2. Demonstration of Economy and Efficiency
8. Have resources in support of the administration of TRIUMF been used in an economic manner?	✓	✓	✓	✓
9. Is TRIUMF administered in an efficient manner?	✓	✓	✓	✓	✓	✓
10. Are there external factors that have an impact on the cost-effectiveness of TRIUMF?	✓	✓	✓
11. Does TRIUMF have the right governance structure in place?	✓	✓	✓	✓	✓

Appendix B: Methodology

The evaluation of NRC's contribution to TRIUMF was conducted to assess the value for money of the program (i.e., relevance and performance) from 2008-09 to 2012-13. As fulfillment of TRIUMF's goals draws upon resources beyond those provided via the Contribution Agreement, the full slate of activities and outcomes of TRIUMF will be considered as part of the evaluation, not only those funded solely by NRC. Other funders (i.e., NSERC, CFI) have been involved in this evaluation through participation as Methodological and Subject Matter Experts (MSEs).

The selection of methods was based upon the most efficient means of addressing the evaluation issues in a rigorous way, while taking into account cost, time and resource constraints, as well as other considerations, such as evaluation scope, evaluation budget, and minimizing response burden. The evaluation approach and level of effort was commensurate with the relatively low level of program risk, including its previous assessments in support of prior funding renewal activities. As such, an objectives-oriented approach was taken, using a non-experimental, largely descriptive evaluation design.

In order to maximize the possibility of generating useful, valid and relevant evaluation findings, mixed methods were used for this evaluation, allowing for triangulation (i.e., convergence of results across lines of evidence) and complementarity (i.e., developing better understanding by exploring different facets of a complex issue). Both qualitative and quantitative methods were used, and include:

• Internal and external document review;
• Administrative and performance data review;
• Key informant interviews (internal and external);
• Survey of users;
• Comparison study of selected international physics facilities; and
• Peer review.

A discussion of the approach used for each of these methods as well as their limitations is provided below. In addition to these methods conducted by the OAE, two studies were commissioned respectively by TRIUMF and NRC to complement this work. TRIUMF commissioned a bibliometric study and the NRC Office of the Vice-President of Emerging Technologies Division commissioned a return on investment (ROI) study, both of which have been incorporated into this evaluation report. Each of these studies has strengths and limitations, which are described within their respective reports.

Internal and External Document Review

Internal and external documents were reviewed, synthesized and integrated into the evaluation to provide context and history, and contributed to the analysis of relevance and performance. The review of internal documents included TRIUMF's strategic and business plans, Terms and Conditions, performance reports, presentations, previous peer-reviews, as well as selected program records (e.g., memoranda to Minister, briefing notes).

Given the relatively low level of risk associated with this program, the evaluation sought, to the extent possible, to minimize the burden on TRIUMF and on the subatomic physics community by leveraging secondary information from a number of independent studies that have been undertaken in recent years, which address similar issues surrounding TRIUMF. These include the two studies recently commissioned by NRC and TRIUMF, the ROI and bibliometric studies, respectively. A selected list of the documents reviewed can be found in Appendix D.

External documents to be reviewed were identified by a variety of sources, including: TRIUMF personnel, NRC Finance Branch, the evaluation's MSEs, other stakeholders, online searches, as well as NRC's Knowledge Management Branch who identified and procured relevant sector information.

Administrative and Performance Data Review

Financial, project and performance data were reviewed and used to develop a program profile as well as answer evaluation questions related to client reach and program performance. At a minimum, the data reviewed cover the period from fiscal years 2008-09 to 2012-13. However, this data may be complemented with data from previous years, where available, in order to ensure appropriate historical context and full coverage of the evaluation issues. Administrative and performance data for 2008-09 to 2012-13 were reviewed to provide information on program outputs and contribute to the analysis of resource utilization (e.g. input costs, expenditures data,) and achievement of expected outcomes (i.e., contracts, international investments), which helped to assess the extent to which TRIUMF had been able to generate economic benefits. Where available and appropriate, administrative and financial data for 2003-04 to 2007-08 were also included. Administrative and performance data were provided by staff at TRIUMF, as well as by corporate branches including the Strategic and Operational Planning (SOP) Branch and Finance Branch.

Key Informant Interviews

Conducting interviews with key informants is an essential element of an evaluation methodology. The information gathered through the qualitative, semi-structured interview process was based on personal experiences, opinions and expert knowledge. This information plays an important role in contextualizing performance data and other statistics.

Interviews were conducted either in-person or by telephone. Each interview lasted between 30 minutes and 1 hour and was conducted using an interview guide. Interview guides provided the common questions to be asked of each interviewee thus ensuring that the same issues were addressed by all relevant interviewees. All interviewees received the interview guide in advance of the interview. In some cases, interviews elected to provide their comments in writing to the evaluation team, either following the interview or in a few cases, instead of participating in an interview.

A total of 35 stakeholders were consulted through the evaluation, including 9 internal stakeholders (NRC management and TRIUMF personnel), as well as 26 external stakeholders (including funders; collaborators; international and Canadian physics facilities; and full, associate and non-member universities). A list of the organizations consulted is included in Appendix E. Interviewees were selected in consultation with the MSEs and TRIUMF management, as well as through a review of TRIUMF's funding sources. To encourage interviewee participation, the Director of TRIUMF sent a memo to all potential interviewees providing an overview of the evaluation and highlighting that they might be contacted to participate as interviewees in support of the evaluation.

User Survey

The survey of TRIUMF's user community was designed to assess questions of relevance, effectiveness, efficiency and economy, and governance. The survey was implemented online and potential respondents were contacted via e-mail. The sample of TRIUMF's user population (n = 760) was limited to those who had active email accounts (n = 708).

The NRC Office of Audit and Evaluation (OAE) designed the survey instrument, in consultation with three MSEs. Once finalized, the survey was programmed into NRC's online survey software (FluidSurveys) by NRC Communications. A pilot version of the survey was sent to seven pre-test participants (five English and two French). Five pre-test participants completed the survey and two participants provided comments (which were incorporated into the final version of the survey).

Survey respondents received an e-mail from TRIUMF's Director inviting them to participate. The survey invitations were sent to the full list of email addresses of identified TRIUMF users. NRC monitored the rate of responses and addressed questions and technical challenges as they arose (e.g. broken links). The survey was open for 20 days. An initial reminder was sent two weeks ahead before the close of the survey to encourage further responses and ensure the achievement of the desired level of response rate. A final reminder was sent the day before the survey closed.

After the closure of the survey, OAE reviewed the results and eliminated outliers and other anomalies. Out of a total population of 760 users, 708 users with active email accounts were emailed the survey, and 250 completed the survey – leading to a response rate of 35.3%.

In terms of limitations, given that the survey was conducted online, it was not possible to control the profile of survey respondents completely. Survey invitations were sent to the email addresses that TRIUMF had on record for each user and it is assumed that the respondent was also the identified user. This limitation was mitigated by sending personalized links that could only be used by one user. Another limitation is that the list of users contained individuals who had not used the facility in the time period of the evaluation (2008 to 2013). This was mitigated by design features in the survey that made the time period clear and by clarifying the scope with enquiring survey recipients.

The survey respondents were made up of approximately 36% Canadian and 61% international users. ^{Footnote 27} They came from the following research areas:

• Nuclear physics (51%);
• Materials science (24%);
• Particle physics (11%);
• Nuclear medicine (10%);
• Accelerator physics (1%); and
• Other/none of the above (3%).

The specific service or area accessed by survey respondents is included in the following table.

Table 9: Breakdown of Survey Respondents by Area of Use/Service Area

Survey question "Over the past five years, I have…"	Number of Survey Respondents
Used TRIUMF's facilities or equipment for research	196
Used TRIUMF's facilities or equipment for my training or education	45
Received professional consultation or services from TRIUMF regarding the design of accelerator components, detectors or experimental equipment	52
Received professional consultation or services from TRIUMF regarding the fabrication of accelerator components, detectors or experimental equipment	42
Received professional consultation or services from TRIUMF regarding the assembly of accelerator components, detectors or experimental equipment	41
Received professional consultation or services from TRIUMF regarding computer networking, intensive data acquisition, processing or storage	38
None of the above	48
TOTAL	250

Source: TRIUMF User Community

Note: Survey respondents could select multiple selections for this question, if applicable. Those who selected none of the above (e.g., those who are currently awaiting access) are not included in analyses relating to their service received.

Comparison Study of Selected International Physics Facilities

A comparison study of large scale subatomic facilities was conducted so as to reveal the relative performance, efficiency, and activity areas of TRIUMF. Additionally metrics (e.g., budget composition, outreach activities, and governance structure) were also collected and compared.

The majority of the data for the study was collected via questionnaires that were sent to nine facilities - of which five participated and returned completed questionnaires. The questionnaire requested quantitative and qualitative data regarding some general areas as well as some customized areas that were adjusted according to the research areas of each facility. Supplementary data was also collected from the facilities' annual reports and websites, where appropriate and available.

The facilities that were selected for inclusion in this study were chosen because of their similarities in one or more research areas that corresponded with TRIUMF's research areas. This focus on research areas, rather than direct facility to facility comparison, was necessary because of identified issues with significant variation in scale, scope, and mandate that was evident among the group of possible comparators.

The list below denotes the facilities, in addition to TRIUMF, that chose to participate in the comparison study. Brookhaven National Laboratory, RIKEN RIBF, Turku Laboratory, and National Superconducting Cyclotron Laboratory (NSCL) were also invited to participate; however, chose not to.

• Fermilab;
• Hevesy-Riso;
• Jefferson Laboratory;
• Paul Scherrer Institute; and
• Rutherford Appleton Laboratory (RAL).

In regards to limitations, the above mentioned issue of variation in scale, scope, and mandate is the most significant. Attempts to mitigate this limitation include the focus on research areas rather than facility level data for most performance metrics (as mentioned above) as well as efforts to normalize performance data (e.g., by dividing it by FTEs in the corresponding research area). However, in regards to the latter, irresolvable concerns were raised by an MSE regarding this approach. To address this, the performance data presented in this report is not normalized in a quantitative way; instead the data is presented alongside qualitative explanations of the significance of the differences in scale, scope, and mandate.

Peer Review

An International Peer Review Committee (IPRC) was convened at TRIUMF November 13-15, 2013 to assess TRIUMF's past performance as well as provide input into the Strategic Plan developed by TRIUMF for their next five years of activity. The IPRC was made up of nine Canadian and international experts, with expertise in each of TRIUMF's main research areas. Potential Committee members were identified and validated through a variety of sources (e.g., NRC, TRIUMF and other subject matter experts) to ensure they had the breadth of knowledge to comment on each of TRIUMF's areas, as well as identify potential biases or conflicts of interest. Committee membership is listed in Appendix F. The NRC Office of Audit and Evaluation (OAE) invited the Committee members and acted as secretariat to the IPRC throughout the peer review process.

The IPRC was provided with a selection of key documents to review (e.g., TRIUMF's proposed Strategic Plan, other TRIUMF documentation, the preliminary findings of the evaluation) and discussed them during a conference call prior to the site visit. The Peer Review site visit took place over 3 days at TRIUMF and included a series of presentations by TRIUMF staff and collaborators to provide the IPRC with the necessary information to respond to the questions posed to them, listed in Table . Committee members had the opportunity to ask for additional information throughout the site visit, and debriefed the TRIUMF leadership team on the last day.

Following the site visit, the IPRC produced a report of their conclusions and recommendations, which was reviewed and endorsed by all members of the Committee and reviewed for factual accuracy by the TRIUMF leadership team. It was then integrated into the evaluation material by OAE to produce the final evaluation report.

Table 10: Peer Review Questions

Relevance
1. What is the role of TRIUMF in supporting Canadian and international scientists and students?
Performance
2. To what extent are TRIUMF's research activities, on a national and international scale, considered leading edge?
3. To what extent has key knowledge been generated as a result of TRIUMF's activities?
4. To what extent has TRIUMF elevated Canada's reputation and international leadership in physics?
5. To what extent is TRIUMF supporting the development and training of students and postdoctoral researchers?
6. How appropriate are TRIUMF's activities related to technology transfer and commercialization? How effective have they been in enabling the Canadian industry to become more competitive?
7. Have there been unintended outcomes (positive or negative) as a result of TRIUMF's activities?
8. To what extent do TRIUMF's governance and management models effectively support its objectives? Are there gaps or redundant elements that inhibit progress or economies?
Future Direction
9. Are the proposed activities included in TRIUMF's 5-year plan appropriate and consistent with the needs and ambitions of the physics community, both in Canada and internationally? Will the plan elevate Canada's reputation and international leadership in nuclear medicine, nuclear physics, materials science, particle physics and accelerators research?
10. Do the requested resources and the laboratory's capabilities give reasonable confidence that the activities of the 5-year plan can be carried out to achieve the stated outcomes?

Appendix C: Logic Model

Inputs	Activities	Outputs	Beneficiaries	Immediate Outcomes	Intermediate Outcomes	Final Outcomes
$44M NRC operating funds $42M other sources Land and buildings (leasing agreements) Equipment and facilities In-house and external researchers and staff Funds and equipment from private-sector partners (partners collaborate with and invest in TRIUMF activities) Organizational and governance structure Government direction and consent	Knowledge creation Conducting research at a global scale on: particle & nuclear physics, nuclear medicine, accelerator S&T, and molecular & materials science Producing particles, beams, and isotopes for research uses Technology services Producing isotopes for clinical and industrial users Providing particle beams for irradiation services Designing and constructing accelerators and detectors Developing business opportunities for physics-based technologies Societal contributions Providing radiation therapy services for specialized eye cancers Producing isotopes for clinical use and commercial sale Providing technical consulting services to Canadian universities, industry and others Providing training for post-secondary students & post-docs in advanced research environment Conducting outreach & science promotion activities Identifying & creating opportunities for international partnerships in science & business	Scientific Discoveries Papers, citations, conferences, invited talks, community-recognition via awards Production of HQP Increased knowledge of high school, undergraduate, & graduate students Post-doctoral fellows, technicians & engineers. Technical Deliverables & Agreements Partnerships: MOU's contracts, collaboration agreements. TRIUMF built detectors and accelerator components delivered to other institutions around the world Isotopes delivered around the world. Technological developments and innovations Economic Stimulus Patents & licenses; spin-off companies; royalties; revenues and sales; business growth for key partners; students visits & interactions; and isotope doses for patients.	Canadian and international researchers, scientists and university students and post-doctoral fellows Canadian medical research organizations Canadian firms involved in nuclear medicine and other isotope-science applications Canadian high-technology SMEs Canadian public, high school students and teachers	International recognition and participation Canadian university researchers use TRIUMF as a hub for international science Global scientists and students have access and use TRIUMF facilities and vice versa Canadian scientists and engineers are highly valued in the area of subatomic physics Canadian companies working with TRIUMF generate economic output and expand exports Attraction and retention of global talent in areas of TRIUMF expertise Societal enhancement Canadian firms supply equipment and apparatus required by TRIUMF and international projects Canadian firms develop and commercialize new products Canadian universities are more competitive in terms of attraction and recruitment HQP Canadian high-school students, teachers, and public understand the role of subatomic physics R&D	Scientific advancement Canadian scientists and researchers participate in and contribute to the worldwide network of subatomic physics facilities located in the major countries of the industrialized world Continued Canadian worldwide excellence in physics & astronomy New knowledge and discoveries stemming from international research collaborations and partnerships Breakthroughs in diagnosing and treating Parkinsons' diseases, other degenerative brain diseases, and cancer Mobilization and circulation of Canadian talent Economic impacts Enhanced global competitiveness resulting in direct and substantial benefits accruing to Canadian companies and the economy Enhanced quality of life for employees/workers in Canadian firms Continued growth of an advanced technology cluster in BC, focusing on particle accelerators and their application to molecular imaging	Improved quality of life and economic growth in Canada through knowledge creation as well as the development and commercialization of new technologies in the physical and life sciences with potential applications in healthcare, environmental sciences, natural resources and engineering

Appendix D: Selection of Documents Reviewed

Bank of Canada. (2013). Monthly Average Exchange Rates: 10 Year Look-Up.

Canada. Parliament. House of Commons. (2010). Leading the Way on Jobs and Growth, Canada's Economic Action Plan Year 2, Budget 2010, James Flaherty.

Canada Foundation for Innovation (CFI). (2012). Major Science Initiatives Oversight Framework.

Canadian Institute of Nuclear Physics. (CINP) (2012). CINP Newsletter April 2012.

Canadian Light Source (CLS). (2013). About Canadian Light Source Inc.

The Council of Canadian Academies (CCA). (2012). The State of Science and Technology in Canada, 2012. Prepared for the Government of Canada.

Government of Canada. (2010). Response to the Report of the Expert Review Panel on Medical Isotope Production.

Halliwell, Janet and Roger Foxall. (2009). Governing and Managing Major Initiatives in S&T – Managing for Success, A Report to the Inter-Agency Working Group on Major Investments in Science and Technology.

Hickling Arthurs Low. (2013). Return on Investment in Large Scale Research Infrastructure, prepared for National Research Council Canada.

Industry Canada. (2007). Mobilizing Science and Technology to Canada's Advantage.

Institute of Particle Physics (IIP). (2013). Director's Presentation at AGM May 2013.

International Peer Review Committee (IPRC). (2013). International Peer Review of TRIUMF: Report of the Committee. Prepared for National Research Council Canada (NRC). December 10, 2013.

KPMG LLP. (2011). The Perimeter Institute for Theoretical Physics. Final Evaluation Report. Ottawa, Ontario.

National Research Council Act, R.S.C. 1985, c. N-15.

Natural Sciences and Engineering Research Council of Canada (NSERC). (2012). 2011-2016 The Subatomic Universe in the Age of Discovery: Report of the NSERC Long-Range Planning Committee.

Natural Sciences and Engineering Research Council of Canada (NSERC). (2006). Perspectives on Subatomic Physics in Canada for 2006-2016. Report of the NSERC Long-Range Planning Committee.

Organisation for Economic Co-operation and Development (OECD). (2012). Graduates by Field of Study.

Science, Technology and Innovation Council (STIC). (2010). State of the Nation 2010: Canada's Science, Technology and Innovation System.

TRIUMF. (2005). TRIUMF User Group – Charter and Bylaws.

TRIUMF. (2008). Scientific Activities Report 2008-2010. Vancouver: Canada.

TRIUMF. (2011). More (Research) with Less (Energy).

TRIUMF. (2012). Business Development Report 2011-2012.

TRIUMF. (2013). Canadian Solution to Medical-Isotope Crisis Demonstrates that Cities Could Produce their own Medical Isotopes.

TRIUMF. TRIUMF Research Topics. (n.d.).

United Nations Educational, Scientific, and Cultural Organization (UNESCO). (2010). UNESCO Science Report 2010: The Current Status of Science around the World.

Appendix E: Organizations Consulted

Internal (9)

• National Research Council (NRC)
• TRIUMF

External (26)

International Subatomic Physics Facilities (4)

• Argonne, USA
• GSI, Germany
• CERN, EU
• KEK/J-PARC, Japan

External Canadian Organizations (22)

• Atomic Energy of Canada Limited (AECL)
• BC Cancer Agency
• Canada Foundation for Innovation (CFI)
• Canadian Association of Physicists
• Canadian Institute of Nuclear Physics
• Canadian Institutes of Health Research (CIHR)
• Carleton University (Full Member)
• Genome BC
• Government of British Columbia
• Industry Canada
• Institute of Particle Physics
• McGill University (Associate Member)
• Natural Resources Canada (NRCan)
• Natural Sciences and Engineering Research Council (NSERC)
• Queen's University (Full Member)
• Science World
• Simon Fraser University (Full Member)
• SNOLAB
• University of British Columbia (Full Member)
• University of Manitoba (Full Member)
• University of Saskatchewan (Non-Member)
• University of Waterloo (Non-Member)

Appendix F: Peer Review Committee Membership

The Peer Review Committee, scheduled to meet at TRIUMF November 13-15, 2013, will be comprised of the following members:

Chair:
Dr. Samuel Aronson
Vice-President, American Physical Society
Brookhaven National Laboratory

Dr. Juha Äystö Director,
Helsinki Institute of Physics

Frenny Bawa
Chief Commercial Officer,
Nanotech Security Corp

Dr. Barbara Jones
Head, Theoretical and Computational Physics
IBM Almaden Research Center

Dr. Silvia Jurisson
Professor of Chemistry & Radiology,
University of Missouri

Dr. Robert McGreevy
Director, ISIS Neutron and Muon Source, Rutherford Appleton Laboratory

Dr. Hugh Montgomery Director,
Thomas Jefferson National Accelerator Facility

Dr. Jerry A. Nolen, Jr.
Argonne Distinguished Fellow,
Argonne National Laboratory

Dr. Maury Tigner
Professor of Physics Emeritus,
Cornell University