Evaluation of the NRC’s Medical Devices Research Centre

Prepared by: Office of Audit and Evaluation, National Research Council Canada

Approval: This report was approved by the NRC's President on September 22, 2019

© 2019 Her Majesty the Queen in Right of Canada, as represented by the National Research Council of Canada.

Cat. No. NR16-298/1-2019E-PDF

ISBN 978-0-660-33076-1

Acronyms

AST	Automotive and Surface Transportation Research Centre
FTE	Full-time equivalent
FWCI	Field-Weighted Citation Index
GBA+	Gender-based analysis plus
GRDI	Genomics Research and Development Initiative
HQP	Highly qualified personnel
MD	Medical Devices Research Centre
MNE	Multinational enterprise
NRC	National Research Council of Canada
OAE	Office of Audit and Evaluation
OGD	Other government department (Canadian federal)
PRC	Peer Review Committee
SME	Small-to-medium enterprise
TRL	Technology Readiness Level

Executive Summary

MD supports Canadian medical device companies, other government departments, and multinational enterprises to develop innovative medical technologies that provide rapid, sensitive, accurate and low-cost solutions aimed at saving lives, reducing healthcare burden and stimulating economic opportunities for Canada. Its research is organized into three thrusts: In Vitro Diagnostics, Implantable Devices, and Simulation and Digital Health. It is the NRC's smallest research centre in terms of both budget and staff.

This evaluation covered 2012-13 to 2017-18, inclusively, and drew on a bibliometric study, case studies of projects with 6 clients, data review, document/literature review, internal and external interviews, and a peer review by experts from industry, government, and academia.

Areas of demonstrated strength

Performance
MD's research was leading edge and contributed to advancements in the medical device field. It also had a positive impact on clients, supporting business innovation and growth of the medical device industry. MD's research is positioned to support government policy solutions in the future.

Uniqueness
There is a need for MD's research, which has unique characteristics for example that differentiate it from other players in the field. MD can play an even greater role in the medical device industry if it were to grow.

Areas for improvement

Appropriateness of research
MD's research made significant contributions to the medical device field. MD can refine the focus of its research within each of its 3 thrusts to have a greater impact.

Engagement
MD has worked with appropriate stakeholders to date. MD can increase its outreach and communication efforts for greater visibility as well as work more with pharmaceutical companies, major diagnostic firms and community health organizations.

Capabilities
MD had the appropriate capacities, competencies and facilities to meet its objectives. There are opportunities for MD to grow in size to have an even greater impact. Heavy dependence on key staff poses a risk to its ability to succeed, should they leave. MD must invest in its current facilities to remain state-of-the-art, which is necessary to continue its research in the future.

Introduction

An evaluation of MD was conducted in 2018-19. It assessed the relevance and performance of the research centre. This reports provides an overview of the main findings and conclusions as well as recommendations for MD.

An evaluation of the MD Research Centre and its Health Technologies program was conducted in 2018-19. The evaluation period covered 2012-13 to 2017-18, inclusively. The evaluation was carried out in accordance with the NRC's approved evaluation plan and Treasury Board policies. MD had not been previously evaluated.

This report begins by providing a profile of MD. It then presents the evaluation findings on MD's uniqueness, appropriateness of research, stakeholder engagement, capabilities, and performance. Following the conclusion are five recommendations for improvements within MD.

Evaluation approach

Methods
In order to maximize the possibility of generating useful, valid and relevant evaluation findings, mixed methods were used. This allowed for convergence of results across lines of evidence and developing a better understanding by exploring different facets of complex issues.

• Bibliometric study (publication citation analyses)
• Case studies (on projects with six clients)
• Data review (administrative and performance data)
• Document/literature review
• Internal and external interviews
• Peer review

For more detailed information on the methods, including challenges and limitations, refer to Appendix A.

Questions
The evaluation questions were developed based on consultations and a review of key documents. The questions were:

1. To what extent has MD scoped its research in the most appropriate areas?
   1. How does the work conducted by MD/HT differ from that conducted by others (e.g., universities)?
   2. Are MD/HT focused on the 'right' areas?
2. Has MD/HT engaged with the right clients, collaborators, and other stakeholders?
3. To what extent does the research centre have the capacities, competencies, and facilities to achieve its objectives?
4. To what extent is MD a leader in scientific excellence in the fields of In vitro diagnostics, implantable devices, and simulation and digital health?
5. Has MD/HT contributed to (or is it positioned to contribute to) the economic growth and prosperity of the Canadian health technologies/medical devices industry?
6. Has MD/HT contributed to (or is it positioned to contribute to) government policy solutions?

Profile

MD supports Canadian medical device companies in their quest for new sources of productivity, competitive advantage and growth by providing customized research and technology solutions. They help their clients develop innovative medical technologies that provide rapid, sensitive, accurate and low-cost solutions aimed at saving lives, reducing the healthcare burden and stimulating economic opportunities for Canada.

Activities and resources

Research activity
MD's research activity has been organized into the Health Technologies program. It has 3 research thrusts:

• In vitro Diagnostics: Develop scalable and affordable bioanalytical solutions that may be deployed in settings such as hospital point-of-care, clinical diagnostics, drug discovery and development, and life sciences applications.
• Implantable Devices: Develop, manufacture, and test biocompatible materials for orthopedic devices.
• Simulation and Digital Health: Offer product development services that address emerging business opportunities in surgical efficiency techniques, medical technology software, health IT and homecare rehabilitation.
• A spectroscopy thrust (In Vivo Imaging) was maintained between 2012-13 and 2016-17 before that area of research was discontinued.

Financial resources
Between 2012-13 and 2017-18, MD had expenditures of $81.0 million and generated $16.7 million in revenues. Revenues were generated from a mix of clients from industry (46%), academia and others (32%), and OGDs (22%). MD is the NRC's smallest research centre in terms of both budget and staff. MD represents 1% of the NRC's total annual budget and 2% of the NRC's total staff.

Revenues and Expenditures ($M)	2012-13	2013-14	2014-15	2015-16	2016-17	2017-18	Total
Revenues	$2.8M	$3.3M	$1.8M	$2.7M	$2.6M	$3.5M	$16.7M
Industry	34%	20%	21%	38%	71%	81%*	46%
Other government departments	36%	23%	25%	17%	20%	12%	22%
Academia and others	30%	57%	50%	45%	9%	7%	32%
Other sources	0%	< 1%	4%	< 1%	< 1%	< 1%	< 1%
Expenditures	$21.8M	$16.0M	$12.3M	$10.4M	$9.6M	$10.9M	$81.0M

*Note: FY2017-18 industry figures includes $730 thousand from IRAP's R&D Certificate Program to offset discount to Canadian SMEs.

The higher costs in 2012-13 to 2014-15 were attributable to:

• satellite locations in Calgary, Halifax, and London, and larger presences in Winnipeg and Ottawa, that were associated with the former institutes that formed MD (and which were subsequently disbanded)
• maintenance of equipment later deemed outside scope of MD (e.g., MRIs under the In Vivo Imaging thrust)

Human resources and facilities

Human resources
MD has a total of 69 staff (as of March 31, 2018) in three locations. It has two sections, BioAnalytical Micro Nano Devices and Simulation and Digital Health, that deliver two of the Health Technologies program's research thrusts – In Vitro Diagnostics and Simulation and Digital Health. The Health Technologies program draws on MD resources for the most part (86% of its labour between 2012-13 and 2017-18 comes from MD). Select staff from the NRC's AST research centre are primarily responsible for implementing the Health Technologies program's third research thrust, Implantable Devices, with support from MD staff. See Appendix C for MD's organizational chart.

Projects and clients/collaborators

MD has completed or implemented 147 projects with 112 unique clients and collaborators from 2012-13 to 2017-18. It has shifted over time from working primarily with academia to industry. Overall, most projects were strategic R&D (63%) though MD also provided technical services, mainly to its industry clients (37%).

Industry including 51 SMEs and 16 MNEs

• 96 projects: 58% R&D, 42% technical services
• $7.7 million in revenues (average revenue/project: $71K)

Academia and others including 16 universities, 13 hospitals and research institutes, and 11 other organizations.

• 39 projects: 74% R&D, 26% technical services.
• $5.3 million in revenues (average revenue/project: $126K)

Other government departments including 4 federal clients (i.e., Canadian Food Inspection Agency, Canadian Space Agency, Defense Research Development, Health Canada)

• 12 projects: 92% R&D, 8% technical services
• $3.6 million in revenues (average revenue/project: $261K)

Uniqueness

Sources: Internal and external interviews, data review, document review, peer review.

MD has unique characteristics that differentiate it from other organizations conducting research in similar areas.

Availability of facilities/equipment
MD's facilities and equipment are maintained by technical officers, as opposed to students, as is the case at universities. This allows for continuity of operations and technical expertise. The presence of a fabrication team also allows MD to customize and build necessary pieces from scratch to provide more complete services than others.

Breadth of expertise
MD has a wide range of multidisciplinary and long standing expertise. As a result, it can work with stakeholders, ranging from academics, to clinicians, to industry.

Absorbs risk
MD has the ability to do early stage research and absorb the risk that industry is unable to or unwilling to.

Technology readiness level
MD generally works within the TRL level 4-6, bridging the gap between early research (1-3) and later development (7-9). MD's work with TRL 4-6 is aligned with the needs of industry. The number of strategic R&D projects MD had increased from three to 23 between 2012-13 and 2017-18, underscoring industry's interest in innovative R&D. The research centre's planned move to focus on lower TRL (1-3) under its most recent strategy (2019 to 2024) is necessary to develop new technologies to stay relevant and provide innovative solutions to industry, which is populated with companies driven by product innovation.

Scale and capacity
MD can go from concept to prototype and therefore work with clients on projects that span the TRL scale.

"MD is professional and very responsive. They are scientifically robust and they understand […] business needs…."—MD Client

MD's collaboration centres will reduce the potential for duplication of effort between the NRC and other organizations conducting research in similar areas.

MD is one of few Canadian organizations conducting research in the areas of Implantable Devices, and Simulation and Digital Health. There are, however, other organizations operating in the area of In Vitro Diagnostics, like MD. Rather than compete, MD is partnering with leading institutions in the field including the Centre Hospitalier Universitaire Sainte-Justine (announced July 2017) and the University of Toronto, to form the Centre for Research and Applications in Fluidics Technology (CRAFT; announced November 2018).

These partnerships will allow MD to:

• leverage complementary know-how, tools and expertise
• strengthen ties with academia
• accelerate the innovation process for the development and commercialization of new technologies and applications in microfluidics
• promote the recruitment and training of post-doctoral fellows and graduate students
• create an innovation ecosystem

"In the microfluidics section, some of the work done is quite unique, very early stage and of significant interest" —External Interviewee

Appropriateness of research

Sources: Document/literature review, internal and external interviews, peer review.

Relevance

Overall, MD's research addressed societal needs as well as areas of importance for the medical device industry. Within each of MD's research thrusts, there are opportunities for further refinement for even greater impact. MD's potential is far greater than what is currently realized, and would require MD to grow.

Disbanded research thrusts
MD's decision to disband its In Vivo Imaging thrust in 2016-17 was appropriate because the domain is saturated with other players.

MD's decision to not pursue a testing and certification thrust was based on two factors: a potential overlap with other organizations in industry, and a lack of relevant competencies within the NRC. There may be an opportunity for the Implantable Devices thrust to collaborate with others (e.g., Health Canada) in this domain.

Current research thrusts
MD's current research is aligned with challenges and opportunities in the medical device sector, including a change in healthcare approach, high import dependence, a need to commercialize research and an aging population.

All 3 of MD's research thrusts have made significant contributions to their field. Their respective focuses are well aligned and based on existing capabilities and strengths of the NRC (i.e., those from previous institutes).

There are opportunities for each of MD's research thrusts to refine its focus in support of even greater impacts. This would also require MD to grow. These include:

• Simulation and Digital Health thrust – focus on digital therapeutics (of which virtual reality is only a small part of), including contactless and visual monitoring of bodily signals, vital signs, and data stream analysis.
• Implantable Devices thrust – have a broader focus that includes support for medical devices in the area of material processing and materials testing for implantable devices.
• In Vitro Diagnostics thrust – further exploit microfluidics and lab-on-a-chip technology.

Further details on the relevance of each thrust are provided on the subsequent pages.

Relevance – Simulation and Digital Health research thrust

Current research
MD's work on interactive simulation addresses a broad range of areas that have importance for industry, as well as the Canadian society and economy. These include:

• medical task simulation (e.g., use of virtual reality for surgical training)
• cognitive care (e.g., use of virtual reality for psychological assessment)
• interactive remote care (e.g., management of remote patients)

Opportunities
The PRC concluded that MD would benefit from re-examining its research strategy and refocusing its research in simulation and digital health in specific areas that try to solve grand challenges. This would enable it to have a greater impact as opposed to the incremental contributions it has had to date. It would also contribute to MD finding the next breakthrough. MD could refocus its efforts in:

• digital therapeutics (of which virtual reality is only a small part of), including contactless and visual monitoring of bodily signals, vital signs, and data stream analysis
• assessments/monitoring of psychiatric and pharmaceutical therapies, which have a bigger market than psychological therapies (as currently targeted by the thrust)

Relevance – Implantable Devices research thrust

Current research
MD's work in implantable devices is aligned with some research areas that have current/future market growth potential and address societal needs. This includes:

• orthopedics (e.g., titanium foam)
• dental implants (e.g., porous coating)

Opportunities
The PRC found that MD needs to think more strategically about the role the Implantable Devices thrust can play in the medical device field, and give it a broader focus than it currently has. This could include:

• support for medical devices in the area of material processing
• materials testing for implantable devices (e.g., in collaboration with Health Canada as part of implantable device certification)

Expansion of the Implantable Devices thrust's mandate would require additional resources. It would, however, allow the Implantable Devices thrust to have a concentrated impact, as opposed to incremental contributions.

Relevance – In Vitro Diagnostics research thrust

Current research
MD's research in microfluidics and lab-on-a-chip are focussed on areas that are important for industry stakeholders, as well as the Canadian economy and society. These include:

• genetic testing (e.g., sample preparation to break the cell and extract nucleic acids where DNA and genes are found)
• infectious disease (e.g., tests for latent tuberculosis and for sepsis)
• oncology (e.g., creation of more sensitive tests in clinical oncology settings)

Opportunities
The PRC concluded that MD's work in the In Vitro Diagnostics thrust has the potential to seed a multi-billion dollar industry for Canada. MD can further exploit its microfluidics and lab-on-a-chip technology. The PRC specifically commented "MD's potential is far greater than what is currently realized. They have microfluidic technology that has applications for MNEs beyond their current SME base. Their expertise is advanced enough and technology sufficiently mature to attract MNE investment and collaboration".

The work of the In Vitro Diagnostics thrust is earlier in its development than the work of the Simulation and Digital Health thrust, so there is naturally greater future potential.

Strategic planning

MD should use a more structured approach and process to identify and prioritize industry needs to ensure continued relevance.

Identification of research priorities
Over the past 5 years, MD has evolved from technology push to using a combination of technology push and market pull when defining its research focus. MD has identified the needs of industry through ongoing interactions with stakeholders and involvement at trade shows and conferences.

The PRC concluded that MD would benefit from re-examining its research strategy, finding that only one of MD's 3 thrusts (In Vitro Diagnostics) had a long-term strategic vision. The In Vitro Diagnostics thrust is focused on addressing large scale problems (e.g., diagnoses of infectious diseases, food safety). The focus of the other two thrusts were more opportunistic in their research priorities, focused on incremental contributions to the field as opposed to addressing big problems. For the Simulation and Digital Health thrust, this was a strategy to validate the next area for which they should focus their efforts on. This also provided a source of revenue while MD worked on other earlier stage research such as its microfluidics and lab-on-a-chip technologies.

Strategic planning process
Across all thrusts, there is limited evidence of a structured and systematic strategic planning process. There would be value in each thrust having its own annual and long-term strategic plan.

Stakeholder engagement plans, intended to structure outreach efforts and identify industry needs, have not been maintained since the HT program's launch in 2015-16.

While there is no evidence that MD missed relevant opportunities as a result of this, it presents a risk for the research centre—that MD will not be aligned with the needs of the medical device industry, affecting its ability to achieve its objectives.

Going forward, the newly created MD Advisory Board (2018-19) can be used to support MD's efforts to identify and prioritize industry needs, and thus inform a new stakeholder engagement plan.

Consideration of end users

When identifying its strategic priorities, MD would benefit from consulting with and giving greater consideration to end users of its technologies. This includes those from diverse (GBA+) populations, consistent with the federal government's priority to consider GBA+ in all of its activities.

There are opportunities for MD to consult with end users and focus/tailor its research and pursuit of technologies accordingly.

There is limited evidence that MD consulted with end users in the design and implementation of its objectives and research. Often MD considered the perspectives of clinicians, viewing them as the end-users of their technology; they did not however consider patients as end users. It is important for MD to consider both populations when planning its research projects.

In addition, the technologies that MD is working on have particular implications for specific groups of people, of which are often diverse populations. It is a federal government priority to consider diverse groups (i.e., GBA+) in all of its activities.

While not a required consideration at the time of MD's inception, there are opportunities for MD to systematically identify and prioritize the needs of specific stakeholders within GBA+, and focus/tailor its pursuit of technologies accordingly.

For example:

• Indigenous communities - MD's work on latent tuberculosis has the potential to address an important need within the indigenous community (tuberculosis has a disproportionate burden in indigenous communities, compared to general population). Likewise, MD's work on Point of Care solutions and interactive remote care has high implications for indigenous populations and/or remote areas.
• People with disabilities - MD's work on orthese hallux valgus (forefoot deformity orthopedics) has implications for people with disabilities.
• Elderly - MD's work on molecular diagnostics and point-of-care testing in the IVD thrust has implications for the effective and convenient diagnosis of old age diseases. Similarly, MD's work on cognitive care and remediation may have implications for the elderly (e.g., managed care implications and triage).

Stakeholder engagement

Sources: Case studies, internal and external interviews, document/literature review, data review, peer review.

Clients and collaborators

MD worked with a number of different types of clients and collaborators, of which the mix is appropriate given its objectives.

MD seeks to catalyze Canada's medical device industry, both by providing research and technology solutions to Canadian SMEs and MNEs with significant investment in Canada, and by working with OGDs to address Government of Canada priorities. The mix of MD's 112 clients is in line with this strategic priority.

The large proportion of industry clients to OGDs is also appropriate, and reflects the relevance of MD within the medical technologies industry ecosystem. Between 2012-13 and 2017-18, the source of MD's revenue shifted from a relative balance of OGD, academia and industry to predominantly industry (81% of revenues in 2017-18 were from industry). Likewise, 60% of MD's clients/collaborators belonged to industry.

Future opportunities in stakeholder engagement

There are additional clients and collaborators that MD could target in order to have a greater impact on the Canadian economy.

MD's work with its current clients and collaborators contributed to its ability to expand its engagement with new organizations and in new areas (e.g., neurosurgery in the cognitive care industry).

In addition, some of the research centre's work resulted in opportunities to increase awareness of MD's capabilities (e.g., use of MD technology at demonstrations provided opportunities for visibility and awareness of its capabilities).

There are additional engagement opportunities for MD to consider, with larger markets for research and the potential for high volume and greater impacts. These include:

• pharmaceutical industry, particularly if MD focuses its Simulation and Digital Health thrust on digital therapeutics, and contactless and visual monitoring of bodily signals/vital signs/data stream analysis
• major diagnostics firms, for MD's In Vitro Diagnostics to further exploit its microfluidics and lab-on-a-chip technologies
• community health organizations, including Indigenous health organizations, which would provide access to large patient populations to facilitate consideration of end users

Awareness of MD

Outside of its specific clients/collaborators, MD is not well known within the broader medical technologies ecosystem.

Barriers

• Limited resources dedicated to business development, outreach and external promotion of MD's work (one Business Advisor, plus MD DG and two Section Heads)
• Confidential nature of some projects that prevents use in presentations/publications
• Limited ability to publish research (due to the NRC context between 2012-13 and 2016-17, where publishing was not viewed as a priority)
• Restrictive government communications policies, and minimal/limited web presence

Recent improvements

• As of November 2018, MD hired dedicated Client Relationship Leaders for each of its 3 thrusts.
• Consistent with NRC-wide changes in 2017-18, there is an increasing emphasis on publications and conference presentations

Opportunities

• Access more resources who are experienced in and dedicated to business development and stakeholder engagement (particularly with MNEs and other top players)
• Embed entrepreneurs within each thrust that can go beyond technology transfer and contract development roles
• Promote more of MD's scientific excellence via publications and participation in conferences, workshops and seminars
• Identify innovative communications strategies, within the government's policies, and use a diversity of tools and instruments (e.g., annual reports that highlight MD's performance)

Capabilities

Sources: Data review, document/literature review, internal interviews, peer review

Capacities and competencies: Current state

MD had appropriate capacities and competences for where it was focused. The complement of staff have a proven track record. Execution of MD's strategy is heavily dependent on Section Heads and key resources, presenting a risk to the research centre.

Demonstrated flexibility despite institutional constraints
MD demonstrated relative flexibility with its staff despite constraints, including

• fewer staff than planned for due to NRC-wide hiring restrictions
• having to build on expertise existing from prior NRC institutes
• retraining some staff from the disbanded imaging thrust (In Vivo Imaging)
• meeting the Health Technologies program objectives while MD staff were also tasked to work on other NRC non-MD programs (e.g., the Printable Electronics program and multiple Aerospace programs).

MD trained staff through internal R&D projects, and by providing staff time for professional development. In 2016-17 and 2017-18, 11% of staff time was on professional development, close to its target of 15%.

Appropriate capacity and competency
MD has appropriate capacities and competencies for the areas where they are focussed.

MD has an excellent staff composition according to the PRC. Several staff have a level of expertise that is at an internationally recognized level, namely in the In Vitro Diagnostics thrust with its current focus on microfluidics. Others are recognized nationally.

Close to half of MD's staff have a PhD. As a result, MD could leverage these staff to have a greater number of adjunct professorships. This would increase MD's ability to draw on students.

Human resource planning
MD has a dependence on Section Heads, with no clear succession plan. This presents a risk to the research centre. Their absence could affect the relevance of MD and its ability to achieve its objectives.

MD's 2019-20 Operational Plan includes the development of a Human Resource Plan. As MD works to develop and execute this plan, it will be important for it to ensure that it has a succession plan for the management team and key resources.

MD's exposure to retirement risk of its workforce is minimal, as MD employees are relatively young compared to the NRC overall. Most MD employees (61%) are aged 45 or younger. Many MD staff highlighted the same two reasons as to why people want to stay at the NRC, despite opportunities in the private sector:

1. Interesting work
2. Flexibility for work-life balance

Capacities and competencies: Growth opportunities

There is an opportunity for MD to grow its operations to have a greater impact in the medical device field.

Additional capacity and competencies
MD needs to add competencies to its staff in order to keep pace with the changing landscape of the medical devices sector and exploit opportunities. According to the PRC, failure to grow is both a lost opportunity and a prescription for eventual obsolescence. In order to exploit new opportunities in all three of its thrusts, additional resources would be needed.

This is particularly the case for the Implantable Devices thrust. While impacts to date have been appropriate given its size, MD can play a bigger role in the field of implantable devices if it were larger and had an expanded mandate.

If Simulation and Digital Health was to focus on digital therapeutics, and contactless and visual monitoring, additional expertise in these areas would need to be developed.

MD itself identified several competency gaps as important going forward. The In Vitro Diagnostics thrust identified gaps in bio-engineering and molecular data analysis, and the Simulation and Digital Health thrust identified gaps in mechatronics, programmers with sector-relevant knowledge (e.g., neuroscience and cognitive fields) and social media expertise to deploy its technology platforms on mobile devices. In addition, increased Technical Officers would allow for quicker turn around on proof of concept and support for maintenance of aging equipment.

Strategies to grow
With the current requirement to maintain or reduce operational expenditures across the NRC, MD's ability to grow is challenging. Collaborations with industry and academia, as well as the collaboration centres, will allow MD to expand its capacity and competencies. The use of students and post-doctoral fellows will also facilitate growth.

MD is also in the process of seeking options to transfer staff with misaligned competencies (due to the closure of a thrust, In Vivo Devices) to other NRC research centres, which will enable MD to hire additional resources with needed competencies.

Facilities

MD's R&D facilities are well organized and appropriate for its needs. However, the equipment in MD's major facility, the BioAnalytical Micro-Devices Clean Room, is in need of renewal to remain state of the art. There is a potential to grow MD's facilities to address promising opportunities.

Capital investments
MD faced challenges accessing capital investment (minor and major) for facility improvements. This is partly due to the elimination of 2 research thrusts and the associated planned capital expenditures, as well as limited spending on major capital (due to the fact MD has not been as successful against other NRC research centres in accessing a limited pool of major capital funding).

Appropriateness of facilities
The In Vitro Diagnostics thrust generally has the equipment it needs within its BioAnalytical Micro-Devices Clean Room. The age of some equipment (i.e., 10-15 years old) is a concern and upgrades are needed to remain state of the art and relevant to stakeholders. The age of some equipment (i.e., 10-15 years old) is a concern and upgrades are needed to remain state of the art and relevant to stakeholders. In 2017-18, the infinity double side mask aligner imploded, which sent Hg vapour into the lab. This safety risk is, however, being mitigated by the research centre. Newer system use laser illumination instead of Hg lamps. The high volume hot embossing system, purchased in 2009-10, was one of the key pieces of equipment that differentiated the NRC from others in the field.

The In Vitro Diagnostics thrust will be able to access newer facilities and equipment through the collaboration centres with the University of Toronto and CHU Sainte-Justine. According to the PRC there would also be significant return on investment for investments in an in vitro diagnostics research and manufacturing hub, which could be achieved through a co-location model of academia, industry, and the NRC in a single building.

The equipment for the Simulation and Digital Health thrust is current and state of the art. It may need more space but this can be achieved through NRC locations in Vancouver that are supporting the Digital Technology cluster.

The Implantable Devices thrust is supported by the Automotive and Surface Transportation Research Centre's Powder Metallurgy facilities. The PRC found these to be state of the art and appropriate for the thrust's needs.

Performance

Sources: Case studies, peer review, bibliometric study, data review, document/literature review, internal interviews

Scientific impact of MD's published research

The scientific impact of MD's published research within academia was not as high as that of the NRC or Canada overall. MD's published research, however, had more of an applied role and was used downstream in patents, medical and treatment guidelines, health policies, and health insurance assessments.

Impact within academia
MD publications between 2012 and 2017 were not cited as much as the NRC and Canadian publications (see Appendix D for data by research thrust).

Impact beyond academia
MD's publications between 2012 and 2017 were cited in more patent documents than the rate for the NRC or Canadian publications. This indicates they are being used in applied work.

Likewise, between 2011-2017, 38 (out of 326) of MD's publications were used outside of academic publications. 3.7% were referenced in Medical guidelines, health benefit programs, or clinical trials, and 8.6% were cited in patent documents. The proportion of MD publications cited in patents, and decision-making and policy-setting documents was similar to that of the Canadian Institute for Health Research (between 2011-2013, 15.2% of MD's publications were cited beyond academia compared to 18% for the Canadian Institute for Health Research). Note: time periods used differ due to methodological considerations; see methodology appendix for further details.

Advancement of scientific knowledge

MD's work was innovative and contributed to advancements in the medical device field. Its current microfluidics and lab-on-a-chip technologies as well as its previous neurosurgical surgical simulators are of international stature.

International stature
MD's current research in microfluidics and lab-on-a-chip are internationally recognized, as well as its previous work on neurosurgical simulators (i.e., NeuroVR).

If MD takes a more strategic direction within its Simulation and Digital Heath and Implantable Device thrusts, refocusing around areas of strength, it has the potential to reach international excellence in these areas as well.

Innovations
MD has an impressive intellectual property portfolio, particularly in microfluidics. Per capita, MD also had the highest rate of innovations within the Life Sciences Division at the NRC.

According to the PRC, the number of unique patents filed and held by MD, as well as its annual intellectual property targets (5 patents per year) could be greater than is currently the case. NRC-wide factors, however, need to be taken into consideration. The patenting process at the NRC may have inhibited researchers willingness to invent. There currently is an NRC initiative to review the inventor award policy and incentive structure.

Advancements to the medical device field
Implantable Device's foam technology led to projects focused on the same technology for use in human health applications.

Simulation and Digital Health's haptic research has helped pushed research in that area forward

In Vitro Diagnostics centrifuge-based assay and use of thermoplastic elastomer work are significant enhancements for that field of science.

Business innovations impacts

MD's work with its clients and collaborators resulted in positive outcomes that will contribute to the growth of the medical device industry, including transferred knowledge and technology, as well as increased product valuation, commercialization and market valuation.

Government solutions

MD is positioned to contribute to government policy solutions, as demonstrated by the alignment of its work with key areas in the areas of health and bio-sciences, and medical devices research and innovation as well as its work with other government departments.

Alignment with federal government policies and priorities
Digital health (e.g., medical device connectivity) is one priority areas in the Health/Bio-sciences Strategy of the Innovation and Skills Plan.

Health technology commercialization of is a priority of the Build in Canada Innovation Program.

MD will support the Digital Technology cluster, aimed at extending digital technology capabilities to an increasing number of industry sectors, including health.

MD's recent move toward focusing on a remote, interactive healthcare platform is aligned with federal priorities on Indigenous Nations and other rural/remote populations.

Past and ongoing collaboration with federal departments with future impacts on health and safety Health Canada/Canadian Food Inspection Agency: Via its In Vitro Diagnostics thrust, MD worked with Health Canada to produce new tools capable of monitoring in real-time presence of Listeria monocytogenes as well as other foodborne pathogens using MD's lab-on-a-chip technology. The Canadian Food Inspection Agency is the end user of this work, and will be able to identify pathogens in contaminated food within 8 hours instead of 5 days. This is to be implemented over the next 5 years.

Communications Security Establishment: Via its Simulation and Digital Health thrust, MD is working with CSE on a whitepaper "Cybersecurity for Medical Devices: Recommended Best Practices During Design, Development and Deployment".

MD has also worked with Defense Research and Development Canada (via the disbanded In Vivo Imaging thrust), and is embarking on a new R&D project with the Canadian Space Agency (via In Vitro Diagnostics).

Conclusions and recommendations

Relevance
MD is addressing a need in the medical device field. While MD's research has focused on areas of importance to the medical device industry, as well as addressed issues of societal value, MD can make adjustments to its research strategy as it goes forward. This includes exploiting its microfluidics and lab-on-a-chip technologies to support the development of a new industry in Canada, expanding the vision/mandate of its Implantable Devices thrust, and refining the focus of the Simulation and Digital Health thrust to digital therapeutics and contactless/visual monitoring of bodily signals. MD's potential is far greater than what is currently realized, and would require MD to grow.

Capacity
While MD's competencies and facilities were appropriate to date, there is an opportunity for MD to grow to have an even greater impact. In addition, key equipment must be renewed to remain state-of-the-art. This is key to MD's relevance and ability to meet stakeholder needs. Over the evaluation period, MD had limited resources dedicated to business development, outreach and external promotion of its work. MD now has additional business development staff. MD's reliance on its Section Heads poses a risk to MD, should these individuals leave. MD could also capitalize on the high educational attainment of its staff by increasing the number that hold adjunct professorships at universities. This would increase access to more students and post-doctoral fellows.

Performance
MD worked with many clients and collaborators. Clients and collaborators reported that MD contributed to the growth of the medical device industry, including increased product valuation, commercialization and market valuation. MD can make itself more known in the medical device ecosystem and expand its stakeholder engagement.

In addition to increased business innovation and positive impacts on clients, MD has contributed to advancements of scientific knowledge in the medical device field. MD's work was innovative and scientifically excellent, attaining international stature in some areas (microfluidics, lab-on-a-chip, neurosurgical simulators). Its publications were cited in patents, treatment guidelines, and health policies. If MD refocuses its Simulation and Digital Health thrust and its Implantable Devices thrust, it can reach international status in these areas in the future.

Finally, MD is positioned to contribute to government policy solutions, as demonstrated by the alignment of its work with federal government priorities as well as its work with other departments.

Recommendations

Management Response and Action Plan

Recommendation	Risk-level Associated with not Addressing Recommendation	Management Response	Measure of Achievements	Proposed Person(s) Responsible	Expected Date of Completion
1. MD should devise and implement a strategic planning process for each of its three research thrusts, and update these plans on an annual basis. Plans should identify opportunities to exploit (e.g., In Vitro Diagnostics' lab-on-a-chip technology), reconsider and revise research focus where appropriate (e.g., Simulation and Digital Health, and Implantable Devices), and consider involvement of and implications for end-users (e.g., patients, diverse populations).	High	Response: Accepted Action 1: Hire Strategic Advisor responsible for, among other things, establishing process and template for Research Thrust Strategic Plans. Action 2: Develop strategic planning process and templates. Action 3: Draft Research Thrust Strategic Plans that are based on a systematic process with consideration of and consultation with stakeholders and end-users like patients and / or diverse populations.	Advisor hired. Strategic planning process and templates developed. Individualized plans for MD Research Thrusts.	Director General, MD	March 2020
2. MD should develop stakeholder engagement plans, per research thrust, and report against progress made to ensure continued alignment with the strategic plan. As part of these stakeholder engagement plans, consideration could be given to the pharmaceutical industry, major diagnostics firms and community health organizations. Engagement plans should be revisited on an annual basis to maintain alignment with the above-recommended strategic plans.	Medium	Response: Accepted Action 1: Draft engagement plans for each thrust and establish a renewal framework.	Stakeholder engagement plans for each Research Thrust.	Director General, MD	June 2020
		Action 2: Report against progress in engagement plans for each thrust.	At least 80% delivery on action items in the stakeholder engagement plans.		June 2021
3. MD should identify strategies to increase awareness of its capabilities within relevant industries, in particular those where its profile is low but there are opportunities for growth (e.g., pharmaceutical industry, major diagnostics firms).	Medium	Response: Accepted Action 1: Draft marketing plan to increase awareness of MD.	Marketing plan in place and being implemented.	Director General, MD	May 2020
4. MD should develop a strategic plan for staff development and succession planning for each of its thrusts.	High	Response: Accepted Action 1: Develop staff development plans for succession planning.	Professional development plan outlined for High potential personnel (HIPO). Initial meetings with individuals identified in the plan to discuss development plans.	Director General, MD	May 2020
5. MD should prioritize investments in its major facility for In Vitro Diagnostics, the Bio Analytical Clean Room, to support current work and to allow for future growth.	High	Response: Accepted Action 1: MD will explore options for increased investment, internally and externally, and seek NRC Senior Executive approval.	Investment options identified. Investment options presented to SEC and/or other group, as appropriate.	Director of Operations, MD	June 2020

Appendices

Appendix A – Methodology

Bibliometric study
The National Science Library's Intelligence and Analytics team conducted a bibliometric assessment of MD's peer-reviewed publications indexed in Scopus for the period 2012–2018 to assess scientific excellence and impact (both within academia and beyond academia). In addition, the NRC's National Science Library collaborated with the Canadian Institute for Health Research to draw on the database they created to assess the use of academic publications beyond academia (e.g., in Medical and treatment guidelines, health policies, health insurance assessments). When looking at impact beyond academia, publications from 2011 were included as it takes time for these publications to have an influence beyond academia (e.g., 7 – 10 years). The time period for comparison with the Canadian Institute for Health Research was limited to 2011-2013 because that was when data was available.

Case studies
In depth case studies were conducted on the impacts of MD's work with six clients. These case studies were chosen to be high impact cases and represented MD's research thrusts, project types, client types and project sizes. The case studies included a review of project documents as well interviews with the MD project lead (s) and representative (s) from the client organization. In total, 16 interviews were conducted - eight with clients and eight with MD project leads. Overall, the case studies covered 12% of MD projects between 2012-13 and 2016-17 and 29% of MD's revenue.

Document and literature review
Internal and external documents were reviewed to provide context and to complement other lines of evidence in assessing relevance and performance.

Data review
Research Centre and program administrative and performance data for 2012-13 to 2016-17 were reviewed to provide information on program inputs (i.e., resources), outputs, and client reach. This included financial data, human resource data, project data and intellectual property data.

Key informant interviews
Interviews were conducted with 25 stakeholders (19 internal and 6 external) to collect information such as personal experiences, opinions and expert knowledge related to the relevance and performance of MD. This information was used to complement other lines of evidence and to contextualize quantitative information.

Peer review
A peer review committee was convened to assess MD along four dimensions: relevance, stakeholder engagement, performance and appropriateness of resources, including capabilities. The Committee was composed of three members plus one chair and included individuals with expertise related to in vitro diagnostics, simulation and digital health, and implantable devices. Members included national and international representatives from academia, government, and industry. Members were expected to participate in the review process in an objective, unbiased and credible manner, with no apparent or perceived conflict of interest. To this end, all members signed a confidentiality and conflict of interest agreement. Each peer review process included:

1. reviewing background material produced by the program and by the NRC evaluation team
2. participating in a pre-site visit teleconference to discuss the Committee's initial assessment of the programs, information gaps and questions.
3. participating in a two and a half day site visit to the NRC

Limitation and mitigation strategies

Availability of data and documents
Valid and reliable data on facility use was not available. As such, the evaluation was not able to assess facility use. In the last two years, MD has, however, begun the process to do so. Likewise, up to date stakeholder engagement plans were not maintained and outreach activities not tracked. As a result, it was difficult to determine whether MD had reached appropriate stakeholders. The views of the peer review committee were used to inform the extent to which MD had reached the right stakeholders.

As a result of changes to MD's Health Technologies program, updated program documentation (e.g., business plan) did not exist. This made it challenging to assess MD's performance against its identified objectives and plans. In order to mitigate this challenge, interviews with internal staff were used to compose an updated profile of MD.

Use of publications to measure excellence
The challenge with bibliometric analysis is that there is a time lag of citation of published work. As a result, the actual use of more recent publications is likely underestimated in the current study. Additionally, not all of MD's research areas publish at the same rate – research in the Simulation and Digital Health thrust is published less given the field it is in - as in computer sciences/software, for which publishing is not the norm. To mitigate this limitation, other lines of evidence were used to assess the excellence and scientific impact of MD's research as well.

Representativeness of case studies
Clients chosen for case studies represented what were expected to demonstrate high impact. There is the possibility that this does not accurately reflect the true impact of the research centre. This, however, is mitigated by the fact that the peer review committee will be exposed to a wider array of MD's work, and will be in a position to comment on whether MD's overall performance is similar to what was observed through the six case studies.

Peer Review Committee members

Mehran Anvari, Chair: Professor, McMaster University; Scientific Director and CEO, Centre for Surgical Invention & Innovation (CSii); Editor-in-Chief, International Journal of Medical Robotics and Computer-Assisted Surgery

Héloïse Côté: Trade Commissioner, Medical Technologies, Global Affairs Canada

Donald Jones: Chairman, Cardiff Ocean Group LLC; Chief Digital Officer, Scripps Translational Science Institute; Advisor/Advisory Board Member, health-related companies and multiple Fortune 1000 Digital Health Advisory Boards

Michael V. Sefton: Professor, University of Toronto; Executive Director, Medicine by Design

Appendix B – List of key documents reviewed

• bccResearch(2018). Virtual Reality Technologies: Global Markets to 2022
• bccResearch ( 2017a). Connected Medical Device Technologies: Global Marketsfor Sensors, Platforms and Processors
• CFID(2019). Pressing Research Priorities, Canadian Foundation for Infectious Diseases
• CIHI (2018). Repeat Hip and Knee Replacements Cost $130 million Annually
• Government of Canada (2018a). Canada's Economic Strategy Tables - Health and Biosciences, The Innovation and Competitiveness Imperative: Seizing Opportunities for Growth
• Government of Canada (2018b). Medical Device Industry Profile
• Government of Canada (2018c). Data Blog - Heart Disease in Canada
• Government of Canada (2018d). Budget 2018- Equality and Growth: A strong Middle Class
• Evans, J. (2018). In Vitro Diagnostics: Technologies and Global Markets, bccResearch
• export.gov. (2018). Canada-Healthcare and Medical Equipment, The US department of Commerce
• Frost and Sullivan (2017a). Advanced Manufacturing TechVision Opportunity Engine: Recent Advances in 3d printing in Medical Applications
• Frost and Sullivan (2017c). Innovations in Surgical Devices, Wound Management, Lab-on-chip Devices, and Cancer Biomarkers- Medical Devices TechVision Opportunity Engine
• Frost and Sullivan ( 2018c). 3D Printing Revolutionizing Medical Device Manufacturing, TechVision Group of Frost & Sullivan
• Halseth, R. and Odulaja, O. (2018). Indigenous Peoples in Canada affected Disproportionately by TB, National Collaborating Centres for Public Health
• Health Technologies Program Business Plan, October 24 2014
• Ma, Y-H, V., Middleton, K. and You, L. (2018). A Review of Microfluidic Approaches for Investigating Cancer Extravasation during Metastasis, Microsystems & Nanoengineering, 4 (17104)
• Marketandmarkets (2018). Microfluidics Market Worth 27.91 Billion USD by 2023
• Market Watch (2018). Cardiovascular Device Market 2018 Global Industry- Key Players, Size, Trends, Opportunities, Growth Analysis and Forecast to 2023
• Mclean, J. and Cribb, R. (2018). Faulty and unproven medical devices implanted in Canadian patients despite known risks
• MD Health Technologies Business Case (Revised), December 13 2013
• MD Health Technologies Program Business Plan, October 24 2014
• MD Health Technologies Program Implementation Plan Summary Final, December 2, 2015
• MD Annual Operational Plan 2017-18 February 8, 2017
• Montalbano, E. (2017). Flexible Glass has Benefits for Nanoscale Medical-Testing Devices
• Ouellet, V., Adhopia, V., and Mckie, D.(2018). 'We're guinea pigs': Canada's oversight process for implanted medical devices stuns suffering patients
• PRNewswire (2017). Global Biomaterials Market Anticipated to Reach $151.65 Billion by 2021, Reports BIS Research
• ResearchandMarkets (2018b). Implantable Biomaterials-Global Market Outlook (2017- 2026)
• Shaw, J. L. V. (2016). Practical Challenges Related to Point of Care Testing, Pract Lab Med
• Statistics Canada (2018). The 10 leading Causes of Death, 2011
• Statistics Canada (2015). Population Projections for Canada, Provinces and Territories
• Stratistics MRC (2018). Implantable Biomaterials-Global Market Outlook (2017- 2026)

Appendix C – Organizational chart