Evaluation of the Metrology Research Centre

Information
Status: Active
Effective date: June 24, 2021
Prepared by: Office of Audit and Evaluation, National Research Council Canada
Approval: NRC's President
ISBN: 978‑0‑660‑39407‑7
Related document: Summary report - Evaluation of the Metrology Research Centre
Alternate format: Evaluation of the Metrology Research Centre (PDF, 1.46 MB)

Acronyms

AI
Artificial Intelligence
AST
Automotive and Surface Transportation
BIPM
Bureau International des Poids et Mesures
CC
Consultative Committee
CETA
Comprehensive Economic and Trade Agreement
CFIA
Canadian Food Inspection Agency
CGPM
General Conference of Weights and Measures
CIPM
International Committee for Weights and Measures
CLAS
Calibration Laboratory Assessment Service
CMC
Calibration and Measurement Capability
CRM
Certified Reference Materials
CSA
CSA Group (formerly known as Canadian Standards Association)
DND
Department of National Defence
ECCC
Environment and Climate Change Canada
EDI
Equity, Diversity and Inclusion
FWCI
Field‑weighted Citation Index
HC / PHAC
Health Canada / Public Health Agency of Canada
HTSN
High Throughput and Secure Networks
IT
Information Technology
IUPAC
International Union of Pure and Applied Chemistry
INMETRO
National Institute of Metrology Standardization and Industrial Quality (NMI Brazil)
IP
Intellectual Property
ISO/IEC
International Standards Organization/International Electrotechnical Commission
IUPAC
International Union of Pure and Applied Chemistry
JCEP
Joint Centre for Extreme Photonics
KC
Key Comparison
MRA
Mutual Recognition Arrangement
MSE
Measurement System Engagement
MSL
Measurement Standards Laboratory (NMI of New Zealand)
MSS
Measurement Science and Standards
NCSLI
NCSL International (formerly known as National Conference of Standards Laboratories)
NIST
National Institute of Standards and Technology (NMI of the USA)
NMI
National Measurement Institute
NMIA
National Measurement Institute – Australia (NMI of Australia)
NPL
National Physical Laboratory (NMI of the United Kingdom)
NRC
National Research Council
NRCan
Natural Resources Canada
nvPM
Non‑volatile Particulate Matter
OGD
Other government department
PIP
Performance Information Profile
PMAR
Performance Measurement and Accountability Reporting
PPE
Personal Protective Equipment
PSES
Public Service Employees Survey
PSC
Public Safety Canada
PTB
Physikalisch‑Technische Bundesanstalt (NMI of Germany)
QM/QMS
Quality Management/Quality Management System
RC
Research Centre
R&D
Research and Development
SC
Supplementary Comparison
SCC
Standards Council of Canada
SI
The International System of Units
SME
Small‑ and medium‑sized enterprise
TC
Transport Canada

Introduction

An evaluation of the Metrology Research Centre was conducted in 2020. It assessed the research centre's relevance and performance. This report provides an overview of the main findings and conclusions as well as recommendations.

The evaluation of the National Research Council (NRC)'s Metrology Research Centre covered the 2014‑15 to 2019‑20 period. It was carried out in accordance with the NRC's approved evaluation plan and Treasury Board policies. The Metrology Research Centre was last evaluated in 2015 as the Measurement Science and Standards (MSS) Portfolio.

This report begins by providing a profile of the Metrology Research Centre. It then presents the evaluation findings on the extent to which the research centre is a leader in scientific excellence; increased Canada's presence and reputation internationally; contributed to advancement of knowledge, government policy solutions, and business innovations; engages stakeholders; has the capacities, competencies, and facilities to achieve its outcomes; and continues to be relevant. Following the conclusion are four recommendations for improvements within the research centre.

Evaluation Approach

Methods

Mixed methods were used to maximize the generation of useful, valid and relevant evaluation findings. This approach also allowed for convergence of results across methods and contributed to a better understanding of complex issues by exploring different facets.

  • Bibliometric study (publication citation analyses)
  • Data analysis (financial, administrative and performance data)
  • Document and literature review
  • Client survey (N=178)
  • Internal and external interviews (N=28)
  • Case studies (N=6)

Two additional methods were incorporated during the evaluation to enhance the evaluation findings.

  • COVID‑19 case study
  • Special study on comparative measures for scientific excellence

For more detailed information on the methods, refer to Appendix A.

Questions

The evaluation questions were developed based on consultations and a review of key documents. The questions were:

  1. To what extent is the Metrology Research Centre a leader in scientific excellence in the areas of calibration and measurement? What key scientific impacts have been achieved? What has enabled those achievements (lessons learned)?
  2. To what extent has the Metrology Research Centre increased Canada's presence and reputation for metrology on the international stage?
  3. To what extent has the Metrology Research Centre contributed to:
    • Advancement of knowledge?
    • Government policy solutions?
    • Business innovations?
  4. Keeping in mind the Metrology Research Centre's legislated mandate, to what extent are they able to engage other stakeholders (internal/ external)? What could be done by the Metrology Research Centre to further expand engagement?
  5. To what extent does the Metrology Research Centre have the capabilities, competencies and facilities to achieve its outcomes?
  6. How does the Metrology Research Centre ensure it continues to be relevant and achieve the highest possible impact? Are there additional areas it should be in?

Evaluation Limitations

Limitations and Mitigation Strategies

Availability of case study interviewees

A number of the client interviews for the case studies did not take place and a number of the case studies had limited associated documentation. The interview period for most of the case studies was during the early stages of the COVID‑19 pandemic and was likely the reason there were so many refusals to participate.

Mitigation

To the extent possible, the case studies were written‑up with documentation substituting for missing interview information and vice versa. However, some of the case studies ultimately had limited information.

Low response rate to the client survey

The NRC evaluators learned during the survey period that clients at the Department of National Defence (DND) and at Defence Research and Development Canada (DRDC) were unable to access the survey, due to their corporate firewall. As well, given the nature of the work of the Metrology Research Centre, for many of the clients the work conducted was very short in duration, and so clients may not have remembered it. For others, the nature of the client relationship was such that they did not view themselves as clients of the research centre (e.g., those purchasing certified reference materials). In fact, 28 clients responded to the survey invitation saying they had never been a client of the Metrology Research Centre. These two factors do not allow the calculation of a response rate, and may have led to a relatively low response rate to the survey.

Mitigation

Where the DND and DRDC individuals contacted the NRC, a unique survey link was provided to enable survey completion, however this happened only for a relatively small number of DND/DRDC clients.

Profile

The Metrology Research Centre is Canada's National Measurement Institute (NMI). In this capacity, it collaborates across the global innovation system to provide metrology research and services that help transform ideas into market‑ready technologies that benefit Canadian society, the economy and the environment. The Metrology Research Centre contributes to the NRC's three main goals: advance scientific excellence, deliver government policy solutions, and support business innovation.

Research Centre Structure

Vice‑President: Emerging Technologies Division

Director General: Metrology Research Centre

Director of R&D / Director of R&D / Director of Operations

Biotoxin Metrology / Black Carbon Metrology / Dimensional Metrology / Electrical Power Measurements / Electrical Standards / Frequency and Time / Inorganic Chemical Metrology / Mass and Related Quantities / Medical and Industrial Dosimetry / Nanoscale Measurement / Organic Chemical Metrology / Photometry and Spectrophotometry / Radiation, Environment and Security / Thermometry and Radiometry / Quality System / Measurement System Engagement / CLAS: Calibration Laboratories Assessment Services

Metrology Advisory Board: Members include representatives from other government departments, Canadian and foreign not‑for‑profit accreditation associations, academia, industry, and the US metrology counterpart

Activities

The Metrology Research Centre's Strategic Framework 2019‑24 identifies three main goals:

Advance scientific excellence by joining the list of the top‑5 NMIs, and attract and retain the best and brightest while striving for an equitable, inclusive and diverse workforce.

Deliver policy solutions by allowing Canadian policy makers to have foresight on emerging needs requiring metrology and standardization (e.g. transport, quantum, advanced manufacturing). In addition, metrology underpins and contributes to new suites of programs.

Support business innovation by contributing to a robust Canadian measurement system, and having influence in key discussions / decisions nationally and globally in such a way that penetration of Canadian goods into international markets is enhanced.

The Metrology Research Centre focuses on providing technical and research services, conducting strategic (or mandate‑related) research, collecting market insights and intelligence, participating in technical working groups, conducting internal dialogue and coordination, and networking and relationship building. These activities contribute to the NRC strategic outcome of improving Canada's socio‑economic prosperity. See Appendix B for the research centre's logic model.

The Metrology Research Centre is a key part of Canada's national measurement system

Canada's national measurement system is comprised of three key players: the NRC (through the Metrology Research Centre), Measurement Canada and the Standards Council of Canada. These core partners deliver on a framework that strengthens competitiveness, supports innovation and international trade, advances the social well‑being of Canadians, and supports new and emerging technologies. Together, these three core organizations serve the measurement needs of a number of federal government departments and agencies, as well as those of other Canadian stakeholders, including the private sector.

Quality Management System (QMS)

The Metrology Research Centre is committed to meeting the expectations of the Mutual Recognition Arrangement drawn up by the International Committee of Weights and Measures (CIPM), as well as the requirements of international standards ISO/IEC 17025:2017 and ISO 17034:2016. A Research Centre Quality System Task Force is in place to ensure these standards are met. The Metrology Research Centre also conducts annual internal audits in all areas. At least once every five years, external peers perform reviews of the QMS that support the research centre's Calibration and Measurements Capabilities (CMCs). The Metrology Research Centre is also engaged in review activities for other metrology institutes.

Source: Document review (Internal documents)

Financial Resources

Revenue

Over the evaluation period, the Metrology Research Centre generated approximately $45 million in earned revenue. Earned revenues remained relatively constant over the evaluation period, fluctuating between a high of $8.2 million and a low of $6.5 million from 2014 to 2020. Revenues from technical services accounted for over half of all revenues. Sales of goods and information products accounts for over one quarter of revenues.

Earned revenues $M

 
Figure 1 - Text version

A horizontal line chart representing the Metrology Research Centre's earned revenue between the years of 2014‑15 and 2019‑20. Different lines represent Total revenue, review from other government departments (OGDs), revenue from Industry, and Other revenue. The values are fairly consistent across the years, and the total revenue stays between $6 and $8.5 million throughout the whole period.

 

Metrology Research Centre's earned revenues per sector of activity for 2014‑15‑2019‑20

  • $23.2 million in revenues from Technical Services (52% of total revenues)
  • $9.9 million in revenues from Strategic Research (22% of total revenues)
  • $11.9 million in revenues from the Sale of Goods and Information Products (26% of total revenues)

The amount of earned revenue from different sources remained fairly consistent over the evaluation period. However, there was an increase in Strategic Research in 2016‑17 of approximately 63% and an increase in Technical Services in 2017‑18 and 2018‑19 of approximately 20% each year. (The large increase in strategic research was primarily driven by two projects that year, with OGDs, amounting to $1.24M.)

Direct Program Expenses

Table 1
Yearly average $25.0 million
Lowest Year $24.2 (2018‑19)
Highest Year $25.9 (2017‑18)

What's the difference between revenue sources?

Strategic research consists of collaborative research projects undertaken with partners to de‑risk R&D and accelerate commercial development timelines or to advance knowledge that is of interest to the NRC. Technical services make use of NRC IP to assist clients in solving immediate technical problems (including maturation of client technology) through the delivery of specialized fee‑for‑service support. The sale of goods and information products include the CRMs sold by the Metrology Research Centre to academic researchers and industry.

Sources: Data review

Human Resources

Workforce

As of March 31, 2020, the Metrology Research Centre employed 176 employees. Of these, 89 were researchers, 63 were technical officers, 11 were in administrative or corporate positions, 4 were in management positions, and 6 were students (3 were unknown). Over the evaluation period, the number of staff in each position has remained fairly consistent.

 
Figure 3 - Text version

Horizontal line chart representing the Metrology Research Centre's FTE count between the years of 2015 and 2020. The line representing researchers remained between 82 and 93 FTEs throughout the whole period. The line representing technical officers remained between 59 and 68 FTEs throughout the whole period. The lines representing admin/corporate, management, and students remained between 0 and 11 FTEs each throughout the whole period.

Number of staff in each position through the years for 2015-20
  2015 2016 2017 2018 2019 2020
Researchers 82 86 92 93 89 89
Technical Officers 61 63 68 69 66 63
Admin / Corporate 8 10 10 8 10 11
Management 5 4 5 6 5 4
Students 3 8 6 0 8 6
 

Equity, diversity and inclusion (EDI)

When compared to workforce availability, the Metrology Research Centre as a whole has good representation of women, however when looking at the distribution of the women across job categories, it is apparent that women are under‑represented in researcher roles. Visible minorities and persons with disabilities are fairly well‑represented in the research centre, and although they are slightly below the labour market availability, they are above the NRC levels.

Table 2
  NRC METRO Labour market availability
Visible minorities 19% 21% 26%
Persons with disabilities 3% 4% 8%

No conclusion can be made regarding Indigenous representation since the number of Indigenous staff at the Metrology Research Centre is five or less; and therefore, not reported due to self‑identification data confidentiality rules.

 
Figure 4 - Text version

Vertical bar graph showing the % of METRO FTEs in various job categories. Metro representation is higher than labour market availability in Administrative, Management, and Technical Officers positions, but it is lower than labour market availability for Researcher positions. The value for labour market availability for researchers is 33%, while the Metro representation is at 21%.

Women representation
  Metro representation Labour market availability
Administrative 88% 78%
Management 40% 36%
Technical officers 37% 30%
Researchers 21% 33%
 

Sources: Data review

Projects and Clients / Collaborators

The Metrology Research Centre had 1,860 unique agreements with clients between 2015‑16 and 2019‑20. Most of the research centre's work (1,700 projects or 91%) focused on the provision of technical services with 61 projects (3%) focused on strategic research and development. The remaining agreements included confidentiality agreements, loan of NRC property, master agreements, and material transfer agreements.

Industry

  • 61% (1,140) of the research centre's clients
  • 41% Canadian and 20% foreign
  • 40% are small‑ and medium‑sized enterprises (SMEs)

Other Government Departments

  • 20% (376) of the research centre's clients
  • 17% Canadian and 3% foreign

Hospitals

  • 9% (168) of the research centre's clients
  • 9% Canadian and <1% foreign

Academia

  • 2% (43) of the research centre's clients
  • 1% Canadian and 1% foreign

Other

  • 7% (135) of the research centre's clients
  • 5% Canadian and 2% foreign
  • Includes laboratories and power companies (e.g., Manitoba Hydro)

Sources: Data review

Scientific Excellence

Overall finding: The Metrology Research Centre has demonstrated scientific excellence through a number of different outputs, including in scientific peer‑reviewed publications and in their calibration services. In certain instances, the Research Centre can be ranked against other NMIs based on counts of these outputs. In these cases, the research centre compares favourably, but context is required around these outputs and rankings (e.g., each CMC is not equal to each other). There is evidence of a number of key scientific impacts achieved by the research centre over the past five years, such as the redefinition of the kilogram and the calibration of radiation dosing or radiation standards. Furthermore, Metrology staff have been recognized for their scientific contributions through the receipt of various national and international awards, academic postings, and invitations to scientific conferences and sessions. Key enablers to these scientific achievements are staff competencies, facilities, and equipment; and national and international relationships and collaborations.

Measuring Excellence

The Metrology Research Centre aims to be a "top‑five" national measurement institute. However, no existing international ranking of NMIs exists, and the fact that the focus of NMIs varies from one country to another, makes it unlikely that a single, internationally‑accepted methodology for ranking NMIs could be developed. In the absence of an international ranking methodology, the evaluation used proxy indicators for assessing excellence, and overall the research centre performs well. Care must be exercised when examining certain individual indicators, but as an aggregate these indicators suggest that the research centre has performed well in its NMI activities. Current work with the NRC's Performance Measurement and Accountability Reporting unit offers an opportunity to develop a methodology to rank the RC against other NMIs.

The Metrology Research Centre's Strategic Plan defines success for the research centre as being a "Top 5 NMI". A special study on comparative measures for scientific excellence was undertaken as part of this evaluation. This study showed that there are a number of important contextual factors to consider when ranking NMIs on any metric. The primary focus of any single NMI varies based on the priority of the country (i.e., serving the country's metrology needs, vs demonstrating leadership in metrology research). As a result, the indicators of importance will differ from country to country. In the absence of an international ranking methodology, the evaluation used a number of proxy measures to assess the excellence of the Metrology Research Centre NMI activities.

While comparison to other NMIs was possible for some of the proxy indicators, there are associated limitations. For example, context is important in some cases, such as when comparing the number of CMCs produced by the NMI - each CMC not being equal and there being different areas of focus for different national measurement institutes, makes the relevance of the Research Centre's ranking, on this indicator, difficult to assess.

The proxy measures used to assess the Metrology Research Centre's NMI activities are presented next. These metrics can be measured against targets set out in the research centre's Performance Information Profile (PIP) and in certain cases to other NMIs.

The Metrology Research Centre is currently working with the NRC's Performance Measurement and Accountability Reporting unit to develop a management toolkit (a replacement to its PIP). This presents an opportunity for the research centre to review, revise, and potentially add metrics that are used to measure its performance, or to develop a methodology for ranking itself against other NMIs.

NMIs can serve two functions: a service function fulfilling the main role of an NMI and the metrology needs of each country (e.g., developing CMCs, Key Comparisons) and a science / research leadership function focusing on publications and leading edge research in the field. The NRC's Metrology Research Centre fulfills both of these functions.

Performance Information Profiles (PIPs) were implemented with the federal government's introduction of the 2016 Policy and Directive on Results. PIPs identify performance indicators and set targets to determine if results are being achieved.

Sources: Bibliometric study, document and literature review, interviews

Scientific Research

The Metrology Research Centre's publications' average Field‑Weighted Citation Index (FWCI) ranks third in comparison to five Research and Technology Organizations (RTOs) selected by the research centre for comparative purposes, but is lower than NRC's and Canada's averages. In comparison to those five RTOs, the Metrology Research Centre excels in the areas of Black Carbon Metrology and Electrical Power Measurement. Metrology staff have been recognized worldwide.

Metrology Research Centre publication statistics are comparable to other NMIs

The Metrology Research Centre had 721 publications between 2014 and 2019. The number of publications per year was fairly consistent (ranging between 99 and 148 publications per year). METRO publications accrued a similar number of views per publication as NRC (23.1, on average). However, the research centre publications' average FWCI of 1.2 was lower than the NRC and Canada averages (1.4 and 1.5 respectively).

 
Figure 5 - Text version

Horizontal bar chart depicting the Metrology Research Centre's publication Field‑Weighted Citation Index in comparison to five other Research and Technology organizations, for the time period 2014‑19. Metrology RC ranks third, with a FWCI of 1.2. It ranks above NPL (UK), with 1.18; MSL (New Zealand), with 1.06; and INMETRO (Brazil), with 0.87; and below NIST (USA), with 1.49; and PTB (Germany), with 1.31.

 

What is the Field‑Weighted Citation Index (FWCI)? The FWCI is a normalized indicator to gauge relative performance of a publication in terms of citations. It takes into consideration the number of publications and publication norms in different research domains.

The Metrology Research Centre's FWCI is above that of the Canadian and Global FWCI in the areas of Black Carbon Metrology and Electrical Standards. The Metrology Research Centre has the highest share of publications in Black Carbon Metrology and Electrical Power Measurement and the highest FWCI in Electrical Power Measurement compared to the five selected RTOs.

Metrology Research Centre staff have received notable awards and recognitions

Since 2014‑15, Metrology staff have been the recipients of various national and international awards, honours, medals, and recognitions, such as the Queen's Jubilee Medals, the Chemical Institute of Canada Medal, the top 118 chemists under 40 awarded by the International Union of Pure and Applied Chemistry (IUPAC), the Gordon E. Moore Medal awarded by the Electrochemical Society, and the Award of Merit awarded by the American Society for Testing and Materials. Furthermore, some research centre staff have been appointed as adjunct university professors, are part of different Consultative Committees (CC) at the CIPM, and have been invited to speak at various scientific sessions and conferences.

Sources: Bibliometric study, document and literature review, interviews

Scientific Research Leadership

The Metrology Research Centre shows scientific research leadership through piloting Key and Supplementary Comparisons and participating in Pilot comparisons.

A standard means of achieving traceability to the International System of Units and demonstrating the equivalency of unit realisation between NMIs, is Key and Supplementary Comparisons. Key comparisons are comparisons of measurement results, identified by expert committees, to examine the principal calibration and measurement techniques in a specific measurement field. In a key comparison, typically, an artefact standard travels between participating NMIs, each lab performs the measurement and provides their results for comparison. The results are analysed to determine a reference value, the overall comparison uncertainty and the degree of equivalence of each NMI's result and associated uncertainty. The Metrology Research Centre's participation in key comparisons is determined by the research centre's CMCs and is based on strategic management decisions on what CMCs are important for NRC and for Canada.

A key comparison is always organised by a pilot lab that prepares the protocol, organises the schedule, analyses the results and prepares the publication. Being chosen as the pilot lab indicates not only the NMI's commitment to the advancement of metrology, but also demonstrates that the lab is well‑recognised as an expert and authority in the community, since the results of the key comparison will impact all NMI's CMCs.

Pilot studies, on the other hand, are usually comparisons designed to validating new methods, protocols or standards. They are part of metrology research and development, and participating in a pilot study demonstrates scientific leadership. Some pilot studies, after successful completion, become Key Comparisons and are expanded to a larger part of the National Measurement Institute community.

Research leadership by national measurement institutes is demonstrated by a number of factors, including participation in key comparisons, acting as the pilot lab in key comparisons, and involvement in the development and execution of pilot studies.

NRC Metrology ranks highly in terms of its participation in key comparisons

As of April 2, 2020, Canada has participated in 217 key comparisons and 31 supplementary comparisons, placing Canada 12th among all countries (106 countries) for participation in key comparisons.

Key Comparisons are organized through the CIPM Consultative Committees or one of the Regional Metrology Organizations, in which case they are linked to a CIPM Consultative Committees Key Comparison.

Supplementary comparisons are normally organized by RMOs to cover regional needs, for instance, measurements of specific artefacts or with lower accuracy measurements. Although outside of the "normal" scope of the Consultative Committees.

Sources: Data review, special study on measures

Scientific Research Leadership

METRO shows scientific research leadership by leading key comparisons

Canada has participated in 19 key comparison pilot studies, placing them 14th among all countries (51 countries have a least one pilot study).

 
Figure 6 - Text version

Horizontal bar chart depicting Canada's participation in key comparison pilot studies, compared with five other countries. Germany ranks number 1 with 126, UK ranks number 2, USA ranks number 3, Canada ranks number 14 with 19, Brazil ranks number 17, and New Zealand ranks number 30.

 
 
Figure 7 - Text version

Horizontal bar chart depicting Canada's rank in percentage of pilot studies out of key comparison studies, compared to five other countries. UK ranks number 1, Germany ranks number 2 (both with 23%), USA ranks number 3 (20%), Canada ranks at 17 and New Zealand at 18 (both with 9%), and Brazil at 36 (with 5%).

 

The percentage of pilot studies out of total key comparison studies is a proxy variable for leadership in metrology‑related activities. In both natural and social sciences, percentage‑based impact metrics are indicators of leadership in a research field.

 
Figure 8 - Text version

Horizontal bar chart depicting NRC's position in serving as the pilot lab for key comparisons, compared to five other institutes. PTB (Germany) ranks number 3 with 82, NIST (USA) ranks number 4, NPL (UK) ranks number 5, NRC (Canada) ranks number 13 with 12, MSL (New Zealand) ranks number 34, and INMETRO (Brazil) ranks number 48.

 

Serving as the pilot lab for a key comparison is an indication of research leadership. The pilot lab is responsible for driving the comparison and analyzing the results of a novel method or standard.

Sources: Data review, special study on measures

National Measurement Institute (NMI) Function

The Metrology Research Centre has produced a number of outputs that fulfill its function as Canada's national measurement institute. Globally, the Metrology Research Centre ranks 14th in total number of Calibration Measurement Capabilities (CMCs) published. The research centre produces CMCs in a number of different areas of metrology.

The Metrology Research Centre serves the NMI function for Canada

The Metrology Research Centre produced:

  • 86 calibration laboratory assessments reports between 2015/16 and 2019/20, for an average of 17.2 per year (target = 15 annually)
  • 102 Certified Reference Materials (CRMs) as of December 2019
  • 24 active licenses (13 initiated since 2014)
  • 549 Calibration and Measurement Capabilities (CMCs) as of April 27, 2020.

The Metrology Research Centre ranks 14th in the world in total CMCs produced and produces CMCs in a breadth of different areas

549 CMCs ranks the Metrology Research Centre as the 14th highest producer of CMCs in the world. Breadth of CMCs in different areas of metrology and impact of each CMC are important factors to consider when counting the total number of CMCs. The research centre produces the fifth highest number of photometry and radiometry CMCs and the tenth highest number of chemistry and biology CMCs in the world.

The number of CMCs in any given area is driven by the technical specificity requirements for different metrology fields, as well as the demand from clients. Several factors can influence demand, including the nature of the industry sector, regulatory requirements, cost, and availability of alternatives.

In some countries Legal Metrology is a part of an NMI, which can greatly skew CMC statistics.

 
Figure 9 - Text version

Horizontal bar chart depicting Canada's position in terms of total calibration measurement capabilities produced, compared with five other countries. USA ranks number 1, with 1,847. Germany ranks number 4, UK ranks number 7, and Brazil ranks number 11. Canada ranks 14th, with 554. And New Zealand ranks 40th.

 

Number of CMCs produced

 
Figure 10 - Text version

Vertical bar chart depicting the number of calibration and measurement capabilities (CMCs) produced by Metrology Research Centre in various areas. Chemistry and Biology is the largest area, with 208 CMCs, followed by Photometry and Radiometry (76); Electricity and Magnetism (73); Thermometry (64); Mass and related quantities (52); Acoustics, Ultrasound and Vibration (32); Length (19); Ionizing radiation (18); and Time and Frequency (7).

 

*Sources: Document and literature review, data review, special study on measures

Scientific Excellence – Examples

The Metrology Research Centre has made significant scientific contributions in a wide range of areas. Key contributions include the redefinition of the kilogram and the calibration of radiation dosing or radiation standards.

Redefining the kilogram

Thanks to the Metrology Research Centre's contribution, in 2017 a value was added to Planck's constant, the base for defining all units within the International System of Units. Since the 19th century, scientists have relied on a physical object to define the fundamental unit of mass – a shining platinum iridium cylinder stored in a locked vault in BIPM in France. Any slight change in the cylinder would lead to a change in the entire global system of measurement. The Metrology Research Centre's contribution helped the international scientific community achieve a more accurate definition of the kilogram (based on a measure of action in quantum mechanics determining mass) which was subsequently adopted by the General Conference of Weights and Measures (CGPM) in November 2018.

Calibration of radiation dosing or radiation standards

The Metrology Research Centre's calibration of radiation dosing or radiation standards is a key scientific contribution useful in a number of areas, including the environment, security, and medical fields. For example, it can be used for such diverse purposes as calibrating medical instruments and scanning a large area for potential dirty bombs.

Sources: Document and literature review, interviews

Enablers of Achievements

Key enablers to the Metrology Research Centre's achievements are the research centre's knowledgeable staff, facilities, and equipment; and national and international relationships and collaborations.

Key enablers

Metrology stakeholders indicate that key enabling factors to achieve these contributions were the research centre's long‑term investments in people and research, including dedicated knowledgeable staff, the necessary equipment, an in‑depth understanding of instruments, and national and international relationships and collaborations. While some NRC interviewees noted challenges associated with turnover of staff, loss of expertise, and aging infrastructure, these concerns have not affected the research centre's ability to achieve successes to date.

  • The expertise of personnel at the Metrology Research Centre is the key advantage the research centre offered over other organizations in Canada and allows the research centre to offer the caliber of service clients are seeking.
  • As described in more detail on the "COVID‑19 Response" section, Metrology Research Centre staff's expertise and flexibility allowed the research centre to respond quickly to the pandemic crisis, enabling them to redeploy capabilities to address priorities of national crises, i.e., quickly construct the testing system for N‑95 respirators.

Sources: Client survey, interviews, case studies, COVID‑19 case study

Contribution

Overall finding: The Metrology Research Centre contributed to the advancement of knowledge through its publications, participation in conferences, working with academia, participating in key comparisons, and as Canada's national measurement institute. The research centre contributed to government policy solutions by collaborating with other government departments and supporting the development of standards and regulations. The research centre contributed to business innovation by providing services to industry that resulted in new products and processes and increased productivity. In addition, the research centre supported commerce as well as the development of trade agreements. The Metrology Research Centre has supported COVID‑19 related activities that have been integral to Canada's response to the pandemic.

Advancement of Knowledge

The Metrology Research Centre is advancing knowledge through: publications; participating in and hosting conferences in Canada; and strengthening the national measurement system.

The Metrology Research Centre advances knowledge through publications

The Metrology Research Centre produced 721 publications (582 journal publications, 136 conference papers, and 3 books and book chapters) between 2014 and 2019. Annually, publications remained fairly consistent over the evaluation period.

The research centre advances knowledge through engagement activities

Since 2014, National Conference of Standards Laboratories International (NCSLI) conferences have been held inside Canada four times, where the Metrology Research Centre offered workshops, technical sessions, exhibitions and tradeshows. Additionally, during 2020‑2021, 57 international conferences were to be held in different areas of metrology. Out of this number, three were to be held in Canada.

  • NRC has decided to move away from hosting conferences, and instead strategically partner with universities and others to co‑host and lead conferences through their institutions / organizations.

In response to a question asking if their organization could access similar products and services from another organization: "Not in Canada. These guys really have world‑class expertise, so I wouldn't be looking elsewhere."

OGD client
The Metrology Research Centre is contributing to a strengthened national measurement system

Clients have identified the Metrology Research Centre as the only organization that can meet their needs, and at a reasonable price. Although other organizations with similar capabilities exist (e.g., companies in Germany and the Netherlands, American laboratories), clients are more likely (81%) to use the Metrology Research Centre than these other providers. Most commonly cited reasons for this are because it is Canada's national measurement institute (49%), because of past experience with the research centre (46%), and because of their trusted research and testing methods (46%). When turning to other organizations, clients most commonly turn to other national measurement institutes, internal resources, and other research and technology organizations, such as the Radiation Safety Institute of Canada, Alberta Innovations, Health Canada, and the Radiological Physics Centre. Industry clients are more likely than government clients to turn to other NMIs and internal resources. In the very few instances where clients go elsewhere, it is because of cost (<1%) and the long process or difficulty of getting an agreement in place with the Research Centre (<1%).

As described in more detail on the "COVID‑19 Response" section, COVID‑19 provided additional opportunities for the research centre to contribute to the national measurement system through the development of respirator testing equipment and disseminating their knowledge and information to others within the federal government, industry, etc.

Sources: Document and literature review, client survey, interviews, COVID‑19 case study

Government Solutions

Other government departments continue to rely on the Metrology Research Centre for support in developing standards and regulations.

The Metrology Research Centre supports Canadian government policy solutions

The Metrology Research Centre collaborates with various federal departments and supports them in their regulatory and standards needs. This includes Public Safety Canada (PSC), Transport Canada (TC), Natural Resources Canada (NRCan), Global Affairs Canada (GAC), Environment and Climate Change Canada (ECCC), the Canadian Food Inspection Agency (CFIA), and Health Canada (HC).

  • In terms of cannabis metrology, the Metrology Research Centre worked with the SCC and PSC to develop and validate a reference method to evaluate potential drug screening equipment for THC (the active component in cannabis), cocaine, and methamphetamine.
  • The research centre's decade‑long collaboration with TC led to the development of the first new regulation for aircraft engine emissions in decades and led to new calibration capabilities in the measurement of Black Carbon allowed for new non‑volatile particulate matter metrology instruments.
  • The Metrology Research Centre supports NRCan in the development of sustainable and value‑added products from the forestry sector. Since 2013‑2014, NRCan has provided over $800,000 in funding for NRC's standardization work.
  • Metrology Research Centre staff, in collaboration with subject matter experts from other countries, published an article on Reference Systems for aircraft engine non‑volatile Particulate Matter (nvPM) emissions measurement in the July 2020 issue of the Journal of Aerosol Science. This study is expected to be important and relevant for researchers, policymakers, and regulators.
  • As described in more detail on the "COVID‑19 Response" section, in response to the COVID‑19 pandemic, the Metrology Research Centre developed unique‑in‑Canada capability to test the performance of N95 respirators and respirator material, worked with HC to develop the standards, and guides for meeting the standards, for respirators and personal protective equipment (PPE), and worked with Standards Council Canada (SCC) to be able to accredit laboratories to the ISO/IEC 17025 standard (the CSA group has received this accreditation).

The research centre also supports small‑ and medium‑sized enterprises (SMEs) to conduct projects supported by OGDs in alignment with Canadian government policy solutions. For example, for a project supported by Sustainable Development Technology Canada, the Metrology Research Centre helped an SME develop the capacity to measure pressure sensors in airplane jet engines to improve the efficiency of the engines.

Sources: Document and literature review, client survey, interviews, case studies, COVID‑19 case study

Business Innovation

The Metrology Research Centre contributed to business innovation, increasing the productivity and competitiveness of industry clients, including supporting new product development. The research centre also had a significant impact on business and international trade through calibration and traceability services as an NMI and by supporting trade agreement negotiations. In response to COVID‑19, the research centre is supporting manufacturers of PPE in Canada by providing product testing as well as guidance and strategic advice.

The Metrology Research Centre has enabled products or processes

The research centre has assisted clients in developing new or improved products and processes (59% of survey respondents noted this). It reduces the uncertainty in clients' measurements and helps them by providing a higher‑quality, less‑expensive service than would be available elsewhere. For example:

  • The research centre has provided key instruments to manufacturers of white products that contain fluorescent‑whiteners (e.g., paper, fabrics, laundry detergents, soaps, plastics, cosmetics)
  • The analysis services provided by the research centre enabled a client to continue to improve its own processes in terms of producing high‑purity materials
  • A technology developed by the Metrology Research Centre is now commercialized and is allowing a Canadian company to offer a competitive solution to the electrical utility industry
  • The research centre supported manufacturers of PPE in Canada during the COVID‑19 pandemic by providing the manufacturers access to testing to confirm the quality of their products and by providing the manufacturers advice and guidance to optimize their products
  • Also related to COVID‑19, the research centre accelerated the development of better performing fabrics for PPE and a new business focus that may not have continued to be pursued had testing taken longer and been more costly to the organization

% clients that agreed the Metrology Research Centre contributed to…

 
Figure 11 - Text version

Horizontal bar graph showing the percentage of surveyed clients who agreed the Metrology Research Centre contributed to the following solutions:

  • New or improved products/solutions: 59%
  • Increased competitiveness: 43%
  • Increased revenues: 35%
  • Improved productivity: 30%
  • Increased export revenues: 22%
  • Reduction is operational costs: 15%
 

The Metrology Research Centre has helped with commerce and trade

The Metrology Research Centre has had a significant impact on business and international trade; for example, through assisting Canada with negotiating the Comprehensive Economic and Trade Agreement (CETA). Further, the research centre is a key enabler of companies' abilities to trade in certain materials because of the credibility associated with the reputable, high‑quality measurement services and the certificates that are provided. This has a direct, positive influence on the competitiveness of companies on the international market for specialty metals and materials.

Sources: Data review, client survey, interviews

COVID‑19 Response

The Metrology Research Centre has been integral to Canada's response to the COVID‑19 pandemic. The research centre has undertaken a number of activities regarding the testing of respirators and expanding those capabilities to other government departments and industry. The research centre has also worked with a number of other partners on respirator testing and other COVID‑19‑related activities.

The Metrology Research Centre's COVID‑19‑related activities included:

  • N‑95 respirator and medical mask testing
  • COVID‑19 protein reference material, with the Human Health Therapeutics (HHT) team
  • Ethanol reference material for sanitizer
  • Blood penetration testing on laboratory coats and personal protective equipment
  • Industrial solvents, in partnership with the Public Health Agency of Canada (PHAC)
  • Advanced thermal imaging for diagnostics

The Metrology Research Centre's activities in the period covering September 2020 to March 2021 primarily involved a shift in focus to the development of respirator and medical mask testing capability.

  • In Phase One, the research centre designed, built, and characterized a testing system using in‑house NRC and partner (e.g., HC, universities) components in response to the unavailability of commercial equipment. The system was built in one week and validated with PHAC the following week. The research centre was the only lab capable of testing respirators for the first six months of the pandemic. As of August 10, 2020, a total of 3,041 N‑95 filtration samples from 607 unique lots of masks were tested by the research centre.
  • Phase Two included consolidating and transferring the knowledge and experience gained in Phase one to create a distributed quality infrastructure to support Canada's developing respirator manufacturing and reprocessing capabilities. As Canadian industry started to delve into the respirator market in response to the pandemic, the Metrology Research Centre helped them by testing their products and providing the evidence needed to gain approval to sell and use the product in Canada. The research centre worked with Health Canada to develop standards and guides for respirator manufacturers to meet the Health Canada requirements. The research centre has done testing for approximately 15 to 20 manufacturers either through the Public Health Agency of Canada, directly through the manufacturers, or through NRC's Industrial Research Assistance Program (IRAP). The research centre has also been working with NRC's Automotive and Surface Transportation Research Centre (AST) to support the manufacturing and reprocessing of PPE with a number of select manufacturers focussing on filtration materials.
  • Phase Three is to solidify national capabilities to create a network of credentialed vendors that can manufacture, reprocess, or test N‑95 respirators. The research centre has been working with laboratories across Canada to develop PPE testing capabilities (e.g., the Vancouver Coastal Health Authority to test the proficiency of respirators, the CSA group in Toronto to validate their respirator testing system and develop a Canadian certification process). The research centre has worked with the Standards Council of Canada (SCC) to be able to accredit laboratories to the ISO/IEC 17025 standard.

Sources: Document and literature review, client survey, interviews, COVID‑19 case study

International Stage

Overall finding: The Metrology Research Centre has a strong reputation and presence internationally. Metrology staff participate in, and chair or lead, a number of important international committees. One of the most significant contributions made by the Metrology Research Centre to international metrology development was redefining the kilogram.

International Stage

The Metrology Research Centre has a strong international reputation among its peers. The research centre has been involved in a number of different areas of metrology and there are examples of the Metrology Research Centre contributing to projects with international recognition.

The Metrology Research Centre plays a key role in Canada's measurement system

The Metrology Research Centre's involvement in Canada measurement system is very important and critical for multiple reasons.

  • The Metrology Research Centre has contributed to the advancement of measurement science.
  • The research centre's place in accreditation cannot be filled by any other organization in Canada.

The Metrology Research Centre has contributed to international metrology‑related developments

The Metrology Research Centre has been involved in international metrology‑related developments in mechanical metrology, electrical metrology, and applying quantum standards in the measurement system.

  • One of the most significant contributions of the Metrology Research Centre to international metrology‑related development is redefining the kilogram based on a measure of action in quantum mechanics determining mass rather than physical objects.

The Metrology Research Centre has a strong international reputation

The Metrology Research Centre is a world leader in its technical expertise and has a strong reputation globally, which is important for the protection of Canada's trade interests, enabling innovation, and enhancing quality of life. Some examples of the research centre's strong international reputation include their involvement in the strontium iron optical frequency standard in collaboration with Canadian universities and European NMIs. Also, publications by research centre scientists in international peer‑reviewed journals further supports its active presence in international metrology activities. The Metrology Research Centre also has a high international collaboration rate when it comes to its publications. Between 2014 and 2019, 75% of the Metrology Research Centre's publications were produced with at least one external partner, including 61% with at least one international partner. Top international collaborative partners were the National Institute of Standards and Technology (NIST) in the USA (n = 41), followed by the Centre national de la recherche scientifique (CNRS) from France (n = 35).

"NRC punches above its weight. At least it has for the last decade or more based upon its influence over the international metrology field"

Expert interviewee

Sources: Document and literature review, interviews, special study on measures

International Engagement

Metrology Research Centre staff is participating in, and leading or chairing, important international metrology committees.

 
Figure 12 - Text version

Vertical bar graph showing the number of Metrology Research Centre staff that are members of important international committees. 30 are Canadian representatives on CIPM committees; 8 have chairing or leading roles; 17 unique individuals are involved in international committees; and there are 47 committees with Canadian representation (including CIPM, SIM).

 
 
Figure 13 - Text version

Horizontal bar graph depicting the Metrology Research Centre's number of staff in CIPM chairing/leading roles compared to four other institutes. Raking first are NIST (USA) and NPL (UK), with 14 staff each in CIPM leading/chairing roles. Third is PTB (Germany), with 12. NRC is fourth, with 8; and NIM (China) is fifth, with 6.

 

As indicated in the chart, the Metrology Research Centre exceeded its performance target (target = 40) in terms of representation on international working groups, scientific committees and/or advisory boards.

Additionally, the research centre has representatives on 9 of the 10 CIPM Consultative Committees and at least 30 CIPM working groups. Other examples of the Metrology Research Centre's international engagement include:

  • The Metrology Research Centre is engaged in at least three technical committees and one task force at the Inter‑American Metrology System (SIM, by its Spanish acronym). Canada hosted three of the SIM council meetings during 2000‑2018.
  • The research centre has a member in Working Group 44 (WG44) at the International Standards Organization responsible for revising ISO/IEC 17025.
  • Metrology Research Centre staff are involved in the ISO/TC 229 with leadership positions, representing the Canadian Mirror Committee.
  • The Metrology Research Centre is an associate member of the Asia Pacific Metrology Programme.
  • Several leadership positions at the International Union of Pure and Applied Chemistry (IUPAC) are affiliated to the NRC.
  • In 2019, the International Year of Periodic Table, a Metrology Research Centre‑affiliated expert co‑chaired the Periodic Table Challenge hosted by IUPAC.
  • Metrology Research Centre experts have also participated in the events or conferences held outside of Canada by the National Conference of Standards Laboratories International (NCSLI).

Sources: Document and literature review, data review, interviews

Stakeholder Engagement

Overall finding: The Metrology Research Centre engages with industry, local and international governments, and academia. Since the previous evaluation, the RC has put in place an advisory board which engages with a variety of stakeholders and is participating in a number of engagement activities, including NRC's Challenge Programs and Cluster Support programs. Clients are satisfied with the level of engagement and services they receive from the research centre. The research centre's involvement in the COVID‑19 response has enhanced its engagement with a number of health and research partners. There may be an opportunity for the research centre to increase awareness of their services. Any additional client engagement effort, however, will need to be strategic, to ensure the research centre can continue to meet its mandate as the country's national measurement institute.

Stakeholder Engagement

Since the last evaluation, the Metrology Research Centre has undertaken a number of initiatives to increase stakeholder engagement with other government departments, industry, and academia.

The 2015 evaluation of the MSS Portfolio recommended more deliberate outreach efforts to engage stakeholders. This need was also emphasized in the Metrology Research Centre's Strategic Framework 2019‑2024. Since that time, the research centre developed two main stakeholder engagement strategies:

  • The "Engagement from the Core" strategy – includes proactive outreach to academia, government science‑based departments, and industrial partners and clients
  • The Measurement System Engagement (MSE) Sector – a Metrology Research Centre‑led knowledge‑sharing platform aimed at ensuring Canada's interests are well‑represented internationally, and providing an avenue for ensuring the research centre is informed of evolving national and international policy priorities

The Metrology Research Centre engages with a number of OGDs and industry clients

The number of research centre projects (i.e., signed agreements) with industry clients has remained constant over the evaluation period, but the number of projects with federal clients has been decreasing since 2015‑16 (40% decrease between 2015‑16 and 2019‑20). However, as noted previously on the "Projects and Clients / Collaborators" section, earned revenue from OGDs has remained constant over the evaluation period, implying that engagement with OGDs has not decreased, but that the projects being undertaken with OGDs are larger and more complex. Also, a potential explanation for the decrease in projects with OGDs is that the research centre project list does not include all collaborations (i.e., only those with official agreements). For example, the cannabis work undertaken by the research centre was a large project that required a lot of resources, and involved collaborating with multiple OGDs, but not all of this work appears in the research centre's list of official agreements.

The Metrology Research Centre most frequently provided services over the past five years to the following federal government departments: Health Canada (HC); the Department of National Defence (DND); Innovation, Science and Economic Development Canada (ISED); Environment and Climate Change Canada (ECCC); the Canadian Food Inspection Agency (CFIA); Transport Canada (TC); the Canadian Nuclear Safety Commission (CNSC); and Natural Resources Canada (NRCan).

The Metrology Research Centre is involved in a number of engagement initiatives

Over the past five years the Metrology Research Centre has become increasingly proactive about engaging with a variety of stakeholders and, in particular, with OGDs, the international community and academia, through other NRC initiatives such as: Challenge Programs (HTSN, and Quantum Sensors), clusters (Oceans, AI for Logistics, and the Digital Health and Geospatial Analytics), the Joint Centre for Extreme Photonics (JCEP), and the Ideation – New Beginnings Initiative. The research centre's involvement in these initiatives has also led to increased engagement with other NRC research centres.

In addition, as previously noted, the research centre's recent role in the COVID‑19 pandemic response has resulted in enhanced relationships with PHAC, HC, SCC, and other research centres at NRC.

Sources: Document and literature review, data review, interviews, COVID‑19 case study

Client Engagement and Awareness

Clients are highly satisfied with Metrology services provided. There may be opportunities to increase awareness of Metrology Research Centre services, although any additional client engagement effort will need to be strategic, to ensure the research centre can continue to meet its mandate as the country's NMI.

 
Figure 14 - Text version

Vertical bar graph depicting clients' satisfaction with Metrology Research Centre's services. 90% would recommend the Research Centre to others, and 93% are satisfied with the services provided.

 

Most often cited as reasons for choosing the Metrology RC:

  • Canada's NMI
  • Past experience with the research centre
  • Trusted research and testing methods
  • Recognized scientific knowledge
  • Research centre reputation

More than half of Metrology's clients (58%, n = 275) undertook more than one project with the research centre between fiscal years 2015/16 and 2019/20 (range = 1 to 52, average = 3.9, median = 2).

Awareness of the Metrology Research Centre's services is low

Awareness is low among research centre clients of the range of services the Metrology Research Centre offers. Clients would like a more active communication strategy on the part of the research centre, and provided suggestions for such things as a newsletter to existing clients, or a bulletin of activities to new and existing clients.

Although the research centre is participating in a number of engagement activities (as noted previously), the Metrology Research Centre does not currently have a strategic client engagement plan and there may be opportunities for increasing awareness of the Metrology Research Centre. However, any new engagement with partners should be strategic and consider the resources the Research Centre has to undertake any new engagements, while still being able to deliver on its NMI function and commitments to previous clients.

"I feel like there's a lot of stuff that goes on at [the Metrology Research Centre] that I don't know about and I would be interested in. I don't know if there's a central newsletter that I can get on."

Client/Collaborator

Sources: Data review, client survey, interviews, case studies

Capabilities

Overall finding: The Metrology Research Centre generally has the needed resources to meet client requests for services. Resources, however, are at their limit, which at times results in project or service delays. Metrology staff have the required competencies to meet client needs and to achieve the research centre's outcomes. The Metrology Research Centre's facilities and equipment are suitable for meeting project objectives and for achieving intended outcomes although there are some concerns about aging infrastructure and limited physical space. The research centre has undertaken initiatives to extend their capital budget by partnering with other organizations and institutions and using minor capital to make gradual improvements to their infrastructure.

Capacity

Metrology staff, although at their capacity limits, are able to undertake their project work and meet client requests. The staffing levels of the Research Centre have remained consistent over the evaluation period, although staff noted challenges related to turnover. Knowledge and skills lost through retirements and other attrition, are not quickly replaced. The research centre has initiatives in place for succession planning.

The Metrology Research Centre has adequate capacity

The Metrology Research Centre has been able to work effectively within the constraints of human and financial resources that are perceived as modest, although stakeholders note that greater impacts could be achieved with increased resources. Metrology researchers were found to be creative in generating as many outputs as possible from existing investment including by leveraging collaborations. To that end, the research centre has the capacity (time in facilities and human resources) to meet clients' project objectives.

Metrology staff have the equipment and materials, and physical environment, needed to complete their work (PSES levels of agreement above Canada and NRC levels), although heavy workloads and a lack of human resources (number of people) were identified, by staff, as issues. The number of staff has remained consistent over the evaluation period, although sub‑optimal human resource levels was an issue raised in the previous evaluation of the research centre as well.

 
Figure 15 - Text version

Vertical bar graph representing the percentage of staff that agreed they have the materials and equipment needed to do their job. In 2017, 88% agreed. In 2019, 91% agreed. The 2019 NRC average was 81%.

 
 
Figure 16 - Text version

Vertical bar graph representing the percentage of staff that agreed they have a suitable physical environment. In 2017, 90% agreed. In 2019, 87% agreed. The 2019 NRC average was 86%.

 
 
Figure 17 - Text version

Vertical bar graph representing the percentage of staff that agreed they are able to complete their work during work hours. In 2017, 58% agreed. In 2018, 54% agreed. In 2019, 63% agreed. The 2019 NRC average was 68%.

 
 
Figure 18 - Text version

Vertical bar graph representing the percentage of staff that agreed that their work suffers due to fewer resources. In 2017, 44% agreed. In 2018, 22% agreed. In 2019, 43% agreed. The 2019 NRC average was 37%.

 

——— 2019 NRC average

Source for charts: PSES

Sources: Data review, PSES, interviews

Effects of limited resources on the Metrology Research Centre

The number of human resources in the Metrology Research Centre has remained consistent over the evaluation period. The volume of work has meant that the research centre has had to prioritize certain projects and initiatives over others. Consequently, it has not always been able to fully seize an opportunity within a given initiative, as might have been desired (e.g. in advanced manufacturing). As well, with respect to client service, some delays have happened, although the research centre tries to mitigate the impact of these delays. However, in certain cases, a lack of human resources or lab space has resulted in negative effects on clients who cannot pre‑book their equipment for calibrations as they wait for instrumentation, samples, etc. For example, clients can wait up to two to three months to be able to schedule a calibration, which results in instrumentation not generating revenue for the client during that period. While some clients have pre‑scheduled calibration times with the research centre to avoid these delays, not all clients are able to do so due to the nature of their business.

The Metrology Research Centre is able to attract staff; new staff face steep learning curves

The Metrology Research Centre has had some success in attracting mid‑ to late‑career researchers from other countries. The research centre also receives a high number of applications from young Canadian researchers, but it was acknowledged that there is a steep learning curve for those young researchers (e.g., five to six years of training is required before someone is fully trained in metrology).

There are concerns regarding the retention of staff

Metrology Research Centre employees feel that the quality of their work suffers because of a perceived lack of stability in NRC and because of high staff turnover (according to the PSES). However, Metrology Research Centre staff are less likely than all NRC staff to indicate that they intend to leave their current position in the next two years, although this proportion has increased by 6% between 2017 and 2019.

There is a perception that the Metrology Research Centre faces challenges in retaining its younger and more recently hired personnel, even though the number of FTEs in all positions has remained fairly consistent and only a few newer technical staff left the research centre during the evaluation period. Metrology Research Centre staff noted the transferable skills of technical officers and suggested there are difficulties competing with attractive salaries offered by university researcher and industry positions. Research officers were generally found to stay with the organization for many years.

Loss of expertise due to attrition is a concern raised by research centre staff, particularly given the length of time needed to fully train someone new. Overlap is needed between senior scientists and more recent hires to increase both the transfer of knowledge and to improve productivity.

Work suffers because of…

 
Figure 19 - Text version

Vertical bar graph representing the percentage of staff that agreed that their work suffered because of a lack of stability in the department. In 2017 and 2018, 56% agreed. In 2019, 62% agreed. The 2019 NRC average was 47%.

 
 
Figure 20 - Text version

Vertical bar graph representing the percentage of staff that agreed that their work suffered from high staff turnover. In 2017, 63% agreed. In 2018, 67% agreed. In 2019, 68% agreed. The 2019 NRC average was 63%.

 

——— 2019 NRC average

Source for charts: PSES

Sources: Document and literature review, data review, PSES, interviews, case studies

Continued improvement of advancement of staff and succession planning

According to the Public Service Employee Survey, only about half of the research centre's employees agree that the NRC does a good job of supporting employee career development and that they have opportunities for promotion within their department.

 
Figure 21 - Text version

Vertical bar graph representing the percentage of staff that agreed that the NRC does a good job of supporting employee career development. In 2017, 60% agreed. In 2018, 56% agreed. In 2019, 59% agreed. The 2019 NRC average was 55%.

 
 
Figure 22 - Text version

Vertical bar graph representing the percentage of staff that agreed that they have opportunities for promotion. In 2017, 55% agreed. In 2018, 43% agreed. In 2019, 53% agreed. The 2019 NRC average was 47%.

 

——— 2019 NRC average

Source for charts: PSES

The Metrology Research Centre has implemented practices for strategic succession planning (in response to the previous evaluation) to address concerns around retirement of key staff. The research centre is training scientific staff to build their management skills through its Director R&D Development Program, which provides leadership and corporate training to team leaders. This program has been positively received to date.

Equity, Diversity and Inclusion (EDI)

While 84% of Metrology Research Centre employees agreed in 2017 that their department implements activities and practices that support a diverse work place, it decreased to about three‑quarters of respondents in 2018 and 2019.

Sources: Document and literature review, data review, PSES, interviews, case studies

Competencies

The Metrology Research Centre has employees that are experts in their fields and can meet the needs of clients and collaborators. The staff is also flexible and able to adjust to new directions in their fields, as demonstrated recently in response to the COVID‑19 pandemic. Staff also receive the training they need.

Competent and adaptable staff

Overall, research centre clients and collaborators agree that Metrology's staff has the requisite competencies and expertise to meet their needs. Staff are considered to be very knowledgeable in their fields of expertise and are professional and responsive to the needs of their clients and collaborators.

Furthermore, Metrology personnel are seen to be able to respond to new directions within their own field or to develop new applications for technologies they already use. Staff expertise in anything measurement‑related allowed the Metrology Research Centre to shift to testing respirators quickly and efficiently in response to the COVID‑19 pandemic.

Sufficient level of training

More than three‑quarters of Metrology Research Centre staff agree that they get the training they need to do their jobs. Both the mentorship and training offered by the research centre are said to be very strong and to be a key factor in the organization's ability to maintain its competency. As well, a variety of training opportunities are available to research centre staff.

Sources: Data review, client survey, interviews, case studies, COVID‑19 case study

Facilities

The Metrology Research Centre facilities are state‑of‑the‑art, but there are concerns about aging infrastructure and limited physical space. The research centre has undertaken initiatives to extend its capital budget by partnering with other organizations and institutions and using minor capital to make gradual improvements to infrastructure.

Metrology's facilities are state‑of‑the‑art, unique, and are for the most part operationally capable, although some current and future concerns exist for certain facilities:

  • Research centre facility reviews* conducted to date, for eight Metrology facilities, concluded that the facilities are state‑of‑the‑art and unique. In terms of operational capabilities, they were considered good for the Photometry and Spectrophotometry, Nanoscale Measurement, and Medical and Industrial Dosimetry facilities. The other reviewed facilities had the following concerns regarding operational capability:
    • The Frequency and Time, Electrical Standards, Dimensional Metrology, and Thermometry and Radiometry facilities have current and future personnel issues due to the retirement of senior research that may cause knowledge gaps.
    • The Electrical Standards, Dimensional Metrology, and Radiation, Environment and Security facilities have potential future physical space issues in terms of aging and adequate infrastructure and the amount of physical space available.
    • The Dimensional Metrology and Thermometry and Radiometry facilities currently have inadequate operational support systems
  • Research centre self‑assessments conducted for the following additional facilities concluded that:
    • The Medical and Industrial Dosimetry and Biotoxin Metrology facilities are state‑of‑the‑art, competitive, unique, and have good operational capability.
    • The Inorganic Chemical Metrology Facility is also state‑of‑the‑art and unique, but could improve in terms of operational capacity with more physical space and improved corporate IT infrastructure.
  • International experts agreed that the Metrology Research Centre facilities are state‑of‑the‑art.
  • Clients and collaborators generally agreed that their needs were being met by the research centre's facilities and equipment.

Concerns about aging infrastructure

The Metrology Research Centre's capital investment spending has been close to the budgeted amount annually during the evaluation period. There is consensus among internal Metrology Research Centre stakeholders that, over the past five years, the NRC has not prioritized major capital allocations; this was noted to extend across NRC and is not limited to the Metrology Research Centre. Stakeholders noted that there is an annual competitive process within the NRC to allocate capital funds across the organization, and based on organizational priorities, the research centre may not have access to all the capital funds desired. While additional funding can always be used to improve facilities or equipment, the Metrology Research Centre manages its available capital responsibly and uses its resources effectively, including using minor capital to make gradual improvements to infrastructure. There is an interrelated nature of infrastructure and personnel, including the role of infrastructure in attracting researchers and the ability of research centre personnel to maintain existing and aging infrastructure.

  • NRC Facility Reviews were conducted by the NRC Finitiatives Team.

Sources: NRC Facility Review reports, Data review, client survey, interviews, case studies

Relevance

Overall finding: The Metrology Research Centre engages with various types of stakeholders (e.g., OGDs, industry) and has an advisory board to ensure the research centre remains relevant. Suggestions for additional areas the Metrology Research Centre should move into were limited. Changes in the field of metrology are slow and the Metrology Research Centre remains relevant by anticipating future client needs and positioning itself to respond to those needs.

Despite the low awareness of clients of the Metrology Research Centre's diverse service offerings, the research centre undertakes a number of activities to ensure it remains relevant to other government departments and industry partners. Research centre staff, clients and collaborators are confident the Metrology Research Centre will remain relevant in the future and continue to meet client needs.

The Metrology Research Centre is undertaking activities to remain relevant

The Metrology Research Centre's process to select projects is influenced by a variety of factors which the research centre seeks to balance, including federal government priorities, NRC and research centre strategic directions, the perceived importance of initiatives, staff expertise and facilities, costs, anticipated beneficiaries and environmental and health impacts. Various processes are used by the research centre to ensure that its activities remain relevant.

  • The Metrology Research Centre regularly gathers insights from a variety of stakeholders, to keep up‑to‑date with evolving priorities, including through post‑project online surveys.
  • The research centre has established various collaborations (e.g. with HC, ECCC, CFIA, DND) to ensure continued relevance to government clients and continues to foster relationships with stakeholders (e.g., through participation in conferences and committees).
  • The research centre's recently launched advisory board, which includes international and national representatives, was identified as an excellent platform
  • At the international level, the research centre's laboratories and personnel undergo rigorous external peer reviews for quality and relevance.
  • As noted previously, Metrology Research Centre staff lead and participate in a number of the CIPM Consultative Committees and CIPM Working Groups which ensures the research centre's and Canada's continued relevance internationally.
  • The research centre remains relevant now (and in the future) by participating and leading developments in the field of metrology, such as their recent commitment to the new CIPM initiative on the Digital System of Units (SI).

The Metrology Research Centre will continue to be relevant in the future

The Metrology Research Centre aligns its activities with future needs and priorities by engaging with other government departments and industry partners to identify their challenges and work to mitigate them. Looking to government priorities can also offer an identification of future needs, as was the case with the legalization of cannabis. Furthermore, the links the research centre has to the international community are a crucial way to determine trends in techniques, types of measurements and refinements.

Internal and external stakeholders agree that the research centre will continue to be able to meet the needs of its clients and collaborators, and that Metrology staff can shift focus as needed in their fields of expertise, as evidenced by their response to the COVID‑19 pandemic.

Few experts, clients or collaborators suggested future areas for the Metrology Research Centre to move into, with the only suggestions being refined infrared wave measurements, radio frequency measurements, and increased consultations on how to design test equipment.

Sources: Document and literature review, client survey, interviews, case studies, COVID‑19 case study

Conclusion and Recommendations

Conclusions

Scientific Excellence

The Metrology Research Centre undertakes a number of service function and research leadership tasks and has demonstrated scientific excellence for a number of different outputs under both functions, including in scientific peer‑reviewed publications and in undertaking calibration services. There is evidence of a number of key scientific impacts achieved by the Metrology Research Centre over the past five years, such as the redefinition of the kilogram and the calibration of radiation dosing or radiation standards. Furthermore, the Metrology Research Centre staff have been recognized for their scientific contributions through the receipt of various national and international awards, academic postings, and invitations to scientific conferences and sessions. However, it was difficult to quantitatively assess the research centre's overall achievement of scientific excellence in certain areas. Although interest in measuring the research centre's excellence against other national measurement institutes was expressed by research centre management, with the exception of a few measures available on the BIPM website, international comparison is difficult.

International Stage

The Metrology Research Centre continues to have a strong reputation and presence internationally. One of the most significant contributions made by the research centre to international metrology development, as identified by international experts, was redefining the kilogram.

Achievement of Outcomes

The Metrology Research Centre continued to contribute to the advancement of knowledge through its publications, participation in conferences, working with academia, participating in key comparisons, and by being Canada's national measurement institute. The research centre contributed to government policy solutions by collaborating with other government departments and supporting the development of standards and regulations.

The research centre contributed to business innovation by providing services to industry that resulted in new products and processes and increased productivity. In addition, the Metrology Research Centre supported commerce as well as the development of trade agreements.

The Metrology Research Centre has also been integral to Canada's response to the COVID‑19 pandemic by undertaking a number of activities regarding the testing of respirators and expanding those capabilities to other government departments and industry, as well as, contributing to other COVID‑19‑related activities. The Metrology Research Centre's contribution to Canada's COVID‑19 response has highlighted the expertise and flexibility of the staff and the leadership role they can play.

Stakeholder Engagement

The Metrology Research Centre engaged with industry, local and international governments, and academia.

Since the last evaluation, the research centre has implemented an advisory board which engages with a variety of stakeholders. It is also participating in a number of recent initiatives, including NRC Challenge Programs and Cluster Support programs. There may be opportunity to increase the awareness of the research centre's services. Any additional client engagement effort, however, will need to be strategic, to ensure the research centre can continue to meet its mandate as the country's national measurement institute.

Capabilities

The Metrology Research Centre generally has the required resources to meet client requests for services. Resources, however, are at their limit, which at times results in project or service delays. Metrology staff have the required competencies to meet client needs and to achieve outcomes. Facilities and equipment are state‑of‑the‑art and suitable for meeting project objectives and for achieving intended outcomes, although there are some concerns related to aging equipment.

Relevance

The Metrology Research Centre engages with a variety of stakeholders and has an advisory board to ensure the research centre remains relevant. Suggestions for additional areas the Metrology Research Centre should move into were limited. Changes in the field of metrology are slow and the research centre continues to remain relevant by anticipating future client needs and positioning itself to respond to those needs.

Recommendations

Recommendation 1

The Metrology Research Centre should develop an approach to collect the data for measures and targets in its management toolkit, that is centralized, readily available, and from which data can be easily extracted.

Rationale: The research centre is tracking performance measures contained in their Performance Information Profile (PIP), however not all data is organized or stored in a way that makes it easy to access and analyze. Since the research centre is currently working on a management toolkit to replace the PIP, there is an opportunity to develop a more appropriate system to collect and organize the data.

Recommendation 2

The Metrology Research Centre should determine an appropriate way to demonstrate its status as a top‑tier NMI aligned with Canadian priorities (i.e., identify the proper metrics and contextual factors to consider, and develop a ranking methodology).

Rationale: Interest in measuring the Metrology Research Centre's excellence against other NMIs was expressed by research centre management, but with the exception of a few measures available on the BIPM website, international comparison is difficult. The special study on comparative measures for scientific excellence undertaken for this evaluation can be used as a base for this exercise.

Recommendation 3

Building on the work done in response to the previous Metrology Research Centre evaluation, the research centre should develop a plan to increase awareness of its research and service activities. The plan should identify appropriate ways to increase client awareness, balanced with the need to be able to continue to meet the research centre's mandate as the country's national measurement institute.

Rationale: The Metrology Research Centre has in recent years undertaken initiatives that are supporting client engagement. However, increasing awareness of the Metrology Research Centre's activities and services has been identified as an area of opportunity. Many external stakeholders have limited knowledge of the breadth of services the research centre provides. Any additional client engagement effort, however, will need to be strategic, to ensure the research centre can continue to meet its mandate as the country's national measurement institute.

Recommendation 4

Given the importance of facility maintenance and upkeep in ensuring the research centre can meet its objectives, and knowing that there will be competing priorities for NRC resources, the Metrology Research Centre should look to increasing its partnerships with other organizations in order to access needed competencies, equipment and facilities.

Rationale: The Metrology Research Centre staff is concerned with aging infrastructure and increasing workloads. Since capital and salary budgets are not likely to increase, the research centre should look at mechanisms to increase their access to state of the art equipment, facilities and human resources. Some cost‑sharing activities have already been undertaken with universities and other government departments.

Management Response and Action Plan

Table 3
Recommendation Risk‑level Associated with not Addressing Recommendation Management Response Measure of Achievements Proposed Person(s) Responsible Expected Date of Completion
1. The Metrology Research Centre should develop an approach to collect the data for measures and targets in its management toolkit, that is centralized, readily available, and from which data can be easily extracted. Low

Response: Accepted

Metrology will establish a process for collecting performance data on a regular basis and a central data repository that is readily available and accessible.

  • The process for data collection is developed, documented, and launched.
  • A centralized accessible and secure location is selected and the repository of outcomes is established.
D‑Ops March 2022
2. The Metrology Research Centre should determine an appropriate way to demonstrate its status as a top‑tier NMI aligned with Canadian priorities (i.e., identify the proper metrics and contextual factors to consider, and develop a ranking methodology). Low

Response: Accepted

Metrology will identify metrics that demonstrate its status as a top‑tier NMI in the world in a manner that accurately reflects its leadership, impact & standing while also demonstrating alignment with Canadian priorities.

Metrics have been defined, including methodology and data sources. Director General, Metrology March 2023
3. Building on the work done in response to the previous Metrology Research Centre evaluation, the research centre should develop a plan to increase awareness of its research and service activities. The plan should identify appropriate ways to increase client awareness, balanced with the need to be able to continue to meet the research centre's mandate as the country's national measurement institute. Medium Response: Accepted      
Action 1: METRO will work with NPBS and Communications Branch partners to develop an updated client engagement strategy while considering RC capacity. Client engagement strategy developed. DRDs March 2022
Action 2: This will include a communication strategy to promote RC services and capabilities among general public via social media and publications. Communication strategy developed resulting in increased number of social media posts & shares, and increased client request for Metrology expertise and services DRDs December 2022
Action 3: METRO will leverage its involvement in NRC challenge and super‑cluster programs to develop new partnerships with academia, OGD's and industry. Partnerships with new collaborators are established. Approved projects within challenge and super cluster programs. Project Manager‑Special Initiatives December 2023
4. Given the importance of facility maintenance and upkeep in ensuring the research centre can meet its objectives, and knowing that there will be competing priorities for NRC resources, the Metrology Research Centre should look to increasing its partnerships with other organizations in order to access needed competencies, equipment and facilities. Low

Response: Accepted

Where appropriate, METRO will continue to explore partnerships to access important but expensive facilities (e.g., 900 MHz NMR in collaboration with uOttawa, Laboratories Canada and OGDs; and Mass Spectrometer - in collaboration with Transport Canada).

In conjunction with recommendation 3, METRO will leverage NRC challenge and super‑cluster programs to strengthen or develop new partnerships with academia, OGD's and industry to access needed competencies, equipment, and facilities.

A process for the ongoing review facilities and capabilities is developed, documented, and launched to ensure that Metrology has access to the facilities required for the delivery of its NMI mandated services and to maintain and improve its Calibration and Measurement Capabilities (CMCs)

DRDs

Project Manager‑Special Initiatives

December 2023

Appendices

Appendix A – Methodology

Document and Literature Review

Internal and external documents were reviewed to provide context and to complement other lines of evidence in assessing relevance and performance. Internal documents included program business plans, research centre operational and strategic plans, presentations, progress reports, peer review reports, facility review reports and assessments, etc. External documents included website information and documents related to government priorities.

Client Survey

External clients and collaborators were invited to complete an emailed client survey which included questions related to the relevance of the research centre activities, satisfaction with the availability of competencies and facilities, and the achievement of outcomes. 178 responded. The breakdown of survey respondents was 45% from industry, 38% from OGDs, 4% from academia, and 12% from other.

COVID‑19 Case Study

A case study specific to the activities undertaken by the Metrology Research Centre in response to the COVID‑19 pandemic between mid‑March and September, 2020 was conducted to provide insight into the research centre's role in Canada's response to the pandemic and the research centre's ability to adapt and change directions as necessary. The case study included a document review and a total of seven interviews, two with research centre staff, three with OGDs, and two with external clients.

Key Informant Interviews

A total of 28 interviews were conducted with stakeholders. This included 19 external clients, partners, and experts and 9 research centre management and staff. This information was used to complement other lines of evidence and to contextualize quantitative information.

Bibliometric Study

The NRC's Intelligence and Analytics unit, KITS, conducted a bibliometric analysis of peer‑reviewed publications affiliated with the research centre and indexed in Scopus for the period 2014‑2019. This analysis was used to answer questions of scientific excellence.

Special Study on Measures

An attempt was made to develop an overall measure of scientific excellence to compare NMIs. This included an in‑depth literature review and interviews with NMI representatives (n = 5). While the methodology for an overall measure has been started, additional work would be needed to complete such a measure.

Data Review

Research centre and program administrative and performance data for 2014‑15 to 2020‑21 were reviewed to provide information on program inputs (i.e., resources), outputs, and client reach. This included financial data, human resource data, project data and client data.

Case Studies

A total of six case studies were completed with industry collaborators/clients (n = 5) and OGDs (n = 1). The case studies included interviews with clients and research centre staff and a document review. This information was used to complement other lines of evidence and to provide examples of client partnerships.

Appendix B – Metrology Research Centre logic model

 
Figure 23 - Text version

Diagram of the Metrology Research Centre Logic Model. The lowest level of the logic model shows the RC Inputs. There are six boxes of inputs: 1. NRC investment; 2. Revenues; 3. NRC common services support and SSC; 4. People, HQP, specialized expertise & experience; 5. Facilities, specialized facilities & equipment; and 6. Contributions from other NRC portfolios & collaborators/partners.

The next level of the logic model has three boxes representing Activities. The first box is entitled "Facilitated scientific, technical & organizational", and includes four activities: 1. Strategic planning (intelligence and foresight), 2. Human resource management (expertise/ workforce development), 3. Facility management, and 4. Program & project management. The second box is entitled "Build scientific & technical capability" and includes three activities: 1. Strategic R&D (internal R&D projects & collaborative R&D projects), 2. Major/minor capital projects, and 3. Participate in technical working groups. The third box is entitled "Deploy scientific & technical capability" and includes two activities: 1. R&D services (collaborative R&D projects and technical services), and 2. Networking & relationship building.

The next level of the logic model has four boxes representing Outputs. The first box is entitled "Infrastructure development" and includes enhanced/maintained infrastructure. The second box is entitled "People development" and includes enhanced/maintained competencies. The third box is entitled "Collaborators" and includes three outputs: 1. Clients/collaborators reached, 2. Partnerships, and 3. Value‑added services & solutions for clients. The forth box is entitled "Scientific/technical achievements", and includes four outputs: 1. New & improved measurement knowledge; 2. Key comparison data, reports & QMS certificates; 3. Measurements; and 4. IP.

The next level of the logic model shows the first level of outcomes of the research centre, the immediate (1 – 5 years) outcomes. There are two boxes of immediate outcomes. The first includes three outcomes: 1. Knowledge of measurement challenges & needs is improved, 2. New & improved measurements standards are refined, and 3. Canada's measurement interests are well represented & supported internationally. The second box includes two outcomes: 1. Public & private sector stakeholders in the national measurement system are supported, and 2. Access to credible measurement services is enhanced.

The next level of the logic model shows intermediate (6 – 10 years) level outcomes. There are three boxes of outcomes at this level. The first box is entitled "Advanced of knowledge" and includes the outcome "The national measurement system is strengthened". The second box is entitled "Government policy solutions" and includes the outcome "National decision‑making for commerce, standards, regulations & trade agreements is informed and improved. The third intermediate outcome box is entitled "Business innovations" and includes three outcomes: 1. Technical barriers to trade are reduced for Canada, 2. Product & process innovation is enabled, and 3. Productivity of clients in target sectors is improved.

The final level of the logic model identifies the research centre's long‑term (10+ years) outcomes. There is just one box entitled "Economic Impact", which includes two outcomes: 1. Ability of vital Canadian sectors to compete, conform and connect on a global stage improved, and 2. Canadian industry ability to capitalize on market opportunities involving emerging technologies is improved.