Evaluation of NRC's Security and Disruptive Technologies Research Centre - Final Report, January 8th, 2018

Key Findings: Need for R&D activities

• The QPSS program is addressing a need in the area of quantum photonics research and development, as quantum technologies are expected to have a significant impact on industries in the future.
• The SMT program is addressing stakeholder needs in the area of advanced materials research and development (R&D), but not to its full potential.

Key Findings: Scientific excellence

• SDTech researchers perform above the world average in terms of the relative impact of their published work, and are recognized nationally and internationally for their research excellence.
• SDTech has been successful in translating its research into promising technologies.
• The QPSS program is conducting world-leading research in the area of attosecond science. In addition, the research conducted in the fields of molecular photonics, quantum dot research and microscopy are considered outstanding.
• The SMT Program is conducting world-leading research in the development and application of boron nitride nanotubes (BNNTs). However, these efforts have not been as visible within the scientific community in the past few years.

Key Findings: Facilities excellence

• For the most part, the QPSS program's quantum photonics research installations are world-class. The lack of access to high performance computing equipment was identified as a weakness.
• The nanotechnology facilities used by the SMT program were found to be comparable to those of other national institutions working in this field of research, with the exception of the production of BNNT s which was considered unique.
• The absence of a scale-up facility is restricting the ability of the SMT program to pursue technology transfer opportunities.

Key Findings: Client and stakeholder engagement and ecosystem development

• While, overall, clients and stakeholder value their relationships with SDTech and reported good alignment with their long term goals, immediate needs and expectations with respect to commercialization have not always been met.
• The level of engagement with quantum stakeholders in Canada was found to be above average in terms of the breadth of stakeholders engaged.
• Concerns were raised regarding the QPSS program's level of engagement with entities that can help bridge the gap between scientific discovery and commercialization.
• While over time the SMT program was expected to increasingly engage industry, the program has not yet achieved this outcome and continues to rely on DRDC as its primary client.
• The SMT program has been successful in developing the Security Materials Technology Roadmap (SMTRM), however, there is limited evidence of implementation beyond the sharing of information.

Conclusions and Recommendations

The evaluation of SDTech revealed that, overall, the Research Centre is, through its two programs, addressing client needs and achieving scientific impacts. Efforts to engage key stakeholders are underway through the Quantum Canada and SMTRM initiatives. Technology development is in the early stages and presents some challenges moving forward, which is not surprising, given that the Research Centre is engaged in activities at low TRL levels.

With respect to each of the programs, the peer review committee provided the following recommendations which have been abbreviated for this executive summary (the full text of the recommendations can be found in Section 4 of the report). In each case, the recommendation is followed by the management response.

2.2 Research Centre Resources

2.2.1 Human Resources

Between 2013 and 2017, the number of R&D personnel^{Footnote 11} within SDTech remained stable at around 65, over two thirds of which have obtained a doctorate. Overall, the Research Centre employs approximately 83^{Footnote 12} staff.

SDTech personnel are divided into seven teams, each led by a team leader. The teams are deployed to support Research Centre programs and the projects within those programs. Areas of specialization are as follows:

• Attosecond Science
• Fibre Photonics
• Nanomaterials Optical and Electronic
• Nanocomposites
• Quantum Theory
• Molecular Photonics
• Quantum Information

In addition to its own resources, SDTech received labour from other Research Centres to supports its activities.

Between FY 2013-14 to 2016-17, over three-quarters of the labour supporting SDTech (78%) was internal (i.e., personnel belonging to SDTech ) and 22% was external to the Research Centre.

Source: NRC financial data system.

Over the period from FY 2013-14 to 201617, SDTech was ranked first among Research Centres in terms of the percentage of labour it received from other Research Centres and second overall (after the Energy, Mining and Environmental (EME) Research Centre) in terms of the percentage of labour it contributed to other Research Centres.

2.2.2 Facilities

SDTech facilities, located in Ottawa, are designed to help increase the understanding of light-matter interactions (used by the QPSS program) and to allow fabrication and prototyping for applications of advanced materials, including nanotechnology (used by the SMT program). Additional information on the facilities used by the QPSS and SMT programs can be found in Section 3.3.

2.2.3 Financial Resources

Between FY 2012-13 and 2016-17, SDTech generated approximately $11.37 Million in revenues, 70% of which was through the provision of strategic research^{Footnote 13} services.

Source: NRC financial data system.

Expenditures associated with SDTech projects totalled approximately $50.6 million between FY 2012-13 and 2016-17. Nearly all these investments (88%) were allocated to strategic research projects.

3.1 Need for R&D activities

Given that Research Centres are aligned with the NRC 's strategic business goals, while programs are more transitory in nature and therefore more subject to change, the evaluation assessed the continued need for the R&D capabilities pursued by SDTech at the program level. The assessment was based on the results of the document review, interviews, a market assessment and the peer review committee's expert opinion.

3.1.1 QPSS Program

The peer review committee (PRC) confirmed that, in quantum photonics, among the biggest technology development needs are in quantum sources and detectors, light-matter interfaces, and quantum memories, all of which are covered by the QPSS program. In addition, important non-linear optics applications are included within the scope of the program.

The PRC further stipulated that, in the future, various sectors of the Canadian economy will need quantum technologies to address their needs. Among the different technologies, quantum photonics offer promising approaches for addressing requirements of three important sectors: ICT , defence and security, and energy/environment. The QPSS program is considered to be well positioned to make significant impacts in these sectors as there is a critical mass of researchers and infrastructures at NRC to address the scientific challenges, and to build unique collaboration with Canadian companies and organizations.

While there are many social benefits associated with and potential commercial applications for quantum technologies, they are new, disruptive, and low- TRL in nature. As a result, R&D in the field carries a certain level of uncertainty and risk, making industry cautious about investing^{Footnote 14}. In these situations the role of government becomes important for filling the gap in private sector investment by supporting basic research and conducting high-risk, low- TRL R&D^{Footnote 15}.

A few internal and external interviewees felt that, while the research is promising, breakthroughs are not expected to occur for another several years, and perhaps not within the timeframe of the program. This was echoed by the PRC who felt that the business and scientific cases are clear, but are definitely long term in nature. As a result, they felt it might be too early to increase the TRL activity in order to achieve an impact on the Canadian economy. The exception is in the areas of resource, environmental and security sensing, where the program has developed technologies to a higher TRL level, which may result in impacts within the program's timeframe.

3.1.2 SMT Program

Addressing deficiencies and technical challenges associated with existing personnel and vehicle armour systems has been identified in government documents as a need by the Canadian military^{Footnote 16}. While the literature acknowledges that existing armour systems are effective (i.e., in terms of stopping threats from projectiles the armour was designed to protect against), studies^{Footnote 17} have underlined a number of challenges associated with existing personal and vehicle armour designs, particularly in the context of military applications. For example:

• A soldier's personal protective equipment (PPE) (or body armour), which typically includes ceramic plate inserts, is heavy. This adds significantly to the total load carried and can restrict mobility. Studies have also shown that heavy loads carried or worn on the body can lead to more rapid fatigue and increased stress on the musculoskeletal system, resulting in higher risk of injury.
• For vehicles, the additional weight of armour can compromise vehicle capabilities, including maneuverability, fuel consumption, transportability, and/or payload capacity.

The SMT program has focused on a specific technological need in the defence sector. Namely, it "was created to support the development of high-performance, cost-effective, and lightweight blast and ballistic protection systems, with a specific focus on advanced personnel protective equipment (PPE) and armoured vehicle (AV) products"^{Footnote 18}.

The PRC found that the activities performed by the SMT program are fully complementary to ones performed at the DRDC (the program's primary client) and there is a clear synergy between their R&D activities. The PRC pointed out that the program has selected specific materials and applications within its business case and has focused on developments in those areas. The arguments made in the foundational program documentation continue to be valid.

In terms of implementation however, the PRC felt the program has not focused sufficiently on addressing the need for fundamental research to widen applications to other areas beyond the defense sector. The current volume of system-level work being conducted with DRDC is low and there is a weak connection between low level TRL research and higher level demonstration because of the lack of a scale-up facility (described in more detail in Section 3.3.2).

3.2 Scientific excellence

Scientific excellence of SDTech as a whole was measured through a bibliometric analysis of publications, awards and prizes, patents and licenses. At the program level, scientific excellence was assessed by the PRC through their review of background materials and participation in the site visit.

3.2.1 Scientific excellence – SDTech

Publications

SDTech as a whole scored above world levels in terms of the quality, scientific impact and excellence of its published works during the period from 2003 to 2014^{Footnote 19}.

At the research sub-field level, SDTech research generally has had a high impact, when compared with the world level during the 2003 to 2014 period.

Source: Science Metrix (2016) Bibliometric Analysis of NRC 's Performance in Emerging Technologies, 2003-2014.

Awards and Prizes

SDTech senior researchers have garnered prestigious awards and prizes from organizations such as the Optical Society of America, the Russian Academy of Sciences, the Royal Society of London and the Royal Society of Chemistry. As information on prizes and awards is not systematically collected by the NRC , it was not possible to provide a quantitative assessment of achievements as part of the evaluation.

Most internal and all external interviewees reported that the quality of research produced by SDTech staff is excellent. They pointed out that SDTech hosts some of the world's leading researchers in the field, conducting innovative and leading-edge research with the support of superior NRC technicians to manage and maintain facilities and equipment.

While the Research Centre's standing in terms of academic excellence is currently high, the profile of the research staff in terms of age and years of service presents a potential risk in the mid- to longer-term due to the significant number of pending retirements. As of May 2017, nearly a third of SDTech R&D personnel (31%) would be eligible for retirement in March 2022 and nearly half (47%) will be eligible to retire by March 2027.

Patents and Licensing

Between FY 2012-13 and 2016-17 SDTech secured a total of 38 patents for 18 different technologies

A total of 5 license agreements for technologies developed by SDTech researchers were secured between FY 2012-13 and 2017-18, including 2 associated with the materials thrust of the Printable Electronic Program

Source: NRC Business Management Services IP data.

3.2.2 Scientific excellence – QPSS Program

The PRC noted that a number of QPSS researchers are recognized nationally and internationally for their excellence. In particular, both the PRC as well as a few internal and external interviewees mentioned Dr. Paul Corkum, who is considered the pioneer of attosecond science. PRC members noted that since he has introduced the 3 step model in 1993, Dr. Corkum has made several pioneering contributions to ultrafast molecular imaging and laser science. Recently, he has extended concepts from attosecond science in gas phase to the solid state. The PRC felt that this will provide novel metrologies for imaging and controlling materials relevant to ICT technologies.

The bibliometric study conducted in 2016 revealed that researchers in the areas of General Physics, and Fluids and Plasmas (most likely supporting QPSS ) have some of the highest impact scores of all SDTech publication areas, further reinforcing the PRC 's view.

In terms of recruitment and retention, the PRC found the QPSS program has been able to attract young promising researchers who have already contributed to the development of emerging quantum technologies including a random number generator. The committee felt the use of volunteer visitors^{Footnote 20} has also allowed the program to continue to capitalize on the expertise of senior scientists. That said, an external interviewee noted that, while the NRC is currently able to attract researchers to the QPSS program because of the reputations of key scientists, this may not be the case once those individuals retire.

3.2.3 Scientific excellence – SMT Program

3.3 Facilities excellence

The evaluation included an assessment of the facilities used by the QPSS and SMT programs, based on interviews with internal and external stakeholders, a review of program documentation, as well as the opinions of the PRC members, based on tours of key facilities, conducted during the site visit.

3.3.1 QPSS facilities

The QPSS program uses NRC R&D facilities located in Ottawa. The facilities are designed to help increase the understanding of light-matter interactions.

The PRC identified a variety of innovative approaches that have been used to create and maintain facilities. In particular, the committee identified the Joint Attosecond Science Laboratory ( JASLab ), the Molecular Science Lab, and the Quantum Security Technology Access Centre (QSTAC) as examples of innovative approaches. Each is briefly described below.

The Joint Attosecond Science Laboratory (JASLab) was created in 2008 through a partnership between the University of Ottawa and the National Research Council. Housed at the NRC 's Sussex Drive facility in Ottawa, it is the only lab in Canada to enable attosecond measurements, with a large variety of end-stations for testing various quantum sensing applications. The partnership between the two organizations was highlighted by the PRC as a novel approach to capitalize on funding opportunities that would otherwise have not been available to the NRC .

The Molecular Photonics Laboratory, also housed at the NRC 's Sussex Drive facility in Ottawa, was built over time through minor capital investments. The PRC highlighted this strategy as being effective in building the lab to the highest standards.

The Quantum Security Technology Access Centre, created in September 2016 at the NRC 's Montreal Road campus in Ottawa, is a joint effort between NRC , the Communications Security Establishment (CSE), and DRDC . The PRC highlighted the fact that QSTAC provides an experimental laboratory for technology testing and demonstration that NRC 's partners were not able to establish themselves.

In addition to the examples above, the PRC also highlighted the cross- NRC collaboration between SDTech and the ICT ^{Footnote 21} Research Centre as an efficient way to secure access to epitaxy and nanofabrication facilities and maximize facility use.

Overall, the PRC found that the JASLab and the Molecular Photonics Laboratory are both world-class facilities providing unique equipment to perform forefront research in quantum photonics. They also identified the QSTAC facility as a promising collaborative effort.

While the PRC was positive in its assessment of most of the facilities, it also cautioned that the lack of access to high performance computing limits the program's ability to do modelling work, particularly within the quantum theory research group. This may have a negative impact on the achievement of results. This caution was echoed by some internal interviewees who specifically mentioned a high reliance on outdated high performance computing equipment, low bandwidth and unstable network connectivity. As well, SDTech operational plans describe issues related to the procurement of new equipment for high performance computing (HPC) capacity. More specifically, the documents indicate that SDTech 's Quantum Theory research group is highly reliant on outdated HPC equipment located at the NRC 's Sussex laboratories.

3.3.2 SMT facilities

The SMT program uses NRC R&D facilities located in Ottawa. The facilities are designed to allow fabrication and prototyping for applications of advanced materials, including nanotechnology (e.g., protective coatings, and ballistic materials).

There was agreement among internal interviewees that while the labs and facilities themselves are not necessarily unique compared to other organizations (with a few exceptions), how the labs are customized and used is unique (e.g., the laboratory for producing BNNT s).

The PRC concurred that the NRC 's nanomaterials facilities are excellent but are equivalent to what is available in a few Canadian universities. That said, the PRC confirmed that the program's material development activities are at the state-of-the-art, as the program is unique in producing BNNT materials in a significant quantity for research purposes.

SMT program documents indicate that a scale-up chemical processing facility is critical to supporting program objectives, particularly under Thrust 1 (technology platform development). According to the SMT business plan, the scale-up facility will be used to produce required amounts of materials to support activities undertaken under program Thrusts 2-4 and to meet the needs of industry as part of technology transfer activities.

In 2015, SDTech submitted a proposal to NRC senior management for approval of $6.9 million in capital investments for the construction of a chemical processing scale-up facility as part of the Flexible Research Facility (FRF) project at NRC 's Montreal Rd. campus. While planning documents indicate that the facility is intended to support SMT , five other NRC Research Centres have expressed interest in participating in the project to support their own activities and programs, including Construction, Aquatic and Crop Resource Development, Energy, Mining and Environment, Aerospace, and Automotive and Surface Transportation. SMT quarterly reports indicate delays in the establishment of the facility, while SDTech 's operational plan for FY 2016-17 indicates that work with NRC Administrative Services and Property Management (ASPM) is ongoing to plan the FRF, including determining the location of the facility and specifications (e.g., designing three units to be used for the new nanomaterial production facility, determining equipment requirements).

Most internal interviewees from SDTech management and the SMT program echoed the concerns raised in the quarterly reports and identified the facility as critical to the success of the program. The PRC agreed that the lack of a scale-up facility for material synthesis and processing is restricting program momentum with respect to technology transfer, noting that without access to a scale-up facility, the expectation of the program should change. At the time of the evaluation the facility had not yet been approved.

Peer Review Committee Member Tasks

The peer review process included three components: the review of the background material collected by the evaluation team, participation in a site visit to the NRC , and the production of a peer review report. The total level of effort required by committee members, including the site visit, was approximately 5 days. As part of the Peer Review Committee, members were responsible for:

• reviewing briefing materials and becoming familiar with the research activities and management practices of SDTech and its programs
• participating in conference calls in advance of the site visit
• attending and actively participating in the peer review process, including the site visit (which occurred from September 26-28, 2017)
• providing input into the peer review report
• reviewing the draft peer review report and providing written comments

Review of Background Materials

Background materials were sent out to peer review committee members for their review prior to the site visit. Members reviewed the materials and completed and "Initial feedback form for Peer Review Committee Members". The purpose of the feedback form was to identify gaps in information that can be filled either through the provision of additional background documents, or through the site visit. The gaps were discussed by the committee during a teleconference call.

Initial Assessment

Peer review committee members then asked to conduct a preliminary assessment of SDTech and its programs, based on the background materials, using an assessment grid created specifically for the evaluation. The assessment grid contained a series of questions and required members to indicate an overall score (using a 3 point scale – below average/average/above average), justification for the score and areas to be further explored during the site visit. A teleconference was held to discuss the assessments and identify key areas to pursue in more detail during the site visit.

Site Visit

The peer review process included a 1.5 day on-site visit to the NRC in Ottawa, Ontario. The visit, which was preceded by a welcome dinner on September 26, occurred on September 27 and 28 and included presentations and discussions on past, current and proposed research activities, and tours of SDTech facilities.

Finalizing the Committee Report

The draft report was formatted and edited following the site visit and additional background information on the purpose and methodology was added to create a complete draft. A report was produced for each of the two programs in addition to an overall evaluation report for SDTech as a whole.

Study Limitations

The following limitations were identified during the course of the project. Where possible, mitigation strategies were used to minimize their impact, as described below.

• Some of the information provided, particularly with respect to bibliometric data, was somewhat outdated. While a more recent bibliometric study was originally expected to be available in time for the review, questions were raised regarding the methodology used, precluding it from being included in the peer review process. In order to minimize the impact, additional information on key researcher accomplishments (awards, prizes) was provided to the peer review committee members.
• Comparisons of the program's performance with that of other similar organizations was not conducted as part of the data collection phase of the project. The peer review committee members' knowledge of other organizations was relied upon to make any necessary comparisons.
• The use of peer review committees can introduce unconscious biases into evaluation studies. In order to mitigate this potential result, efforts were made to ensure members were drawn from a variety of organizations and geographic regions. Significant effort was also made to ensure female representation on the committee.
• Only a limited number of external interviews could be conducted during the timeframe of the project. Efforts were made to triangulate information across several lines of evidence in order to provide summary information to the peer review committee.
• The length of the site visit was limited to 1.5 days (with a working dinner the night before) in order to minimize the burden on committee members and work within budget limitations. In order to minimize the impact, committee members were provided with materials prior to the site visit and responses to questions arising from their initial review of the materials were provided in writing prior to the site visit.